AN 
EVALUATION ASSESSMENT 
OF THE 
SERVICE BUREAUX PILOT PROGRAM 


—_. 
————— 
—sses 
— 
—Ssae 
—_——— 
es 
——————| 
Sass 
————— 
——e 
= 
——____ 
eet 
as 
a 


3 1761 11709885 5 


INN 


The DPA. rouge ite. 


f 1 A Bote . a : ral G PROT a SN: 
7 ‘ \ | H eee fie ¥ feel A: 
§ {i \ ; i we t ‘ $. AY be i. + k : 4 ane 
I Paige ml oh Ki tall 
§ i Lbs 
Bit j , 
it ] } 
fl shin Ati Hepa 
; Tale haere | 
t Hf ; 
i. 4 f 
Ht Tit ] 
: ising teatt ; 
I i 1 pol 7 
( Cento 
; ; Hips rane Pat / ‘ 5 Nr 0 Riribisie Cua Ya ¢ ; : c 
i " ' rats eet ipo eetertiet f Hs y t ; é : 
} PSE Mo ated a TG hea i eat i ; Peet Ss ; : SH 
i Hist ft G i itaol ti i) 1H ae \ fens ' ‘ 
{ ant i ear inian inert} ‘ ; Val t ; Beit Wiad Bee sits f : facies 
} i i fri : s ewe : nse ae psa Mi : , l 
i " 4 ‘ t : VeEln H a i “f A ay HES OAT Tents WR Te { 4 ) , bes ’ 
t i F if 7 red * n ic 
; pide i mares 1! rly } ’ ; 
ities Bist f nu i te 
. : he i ul 4 : See ey i y 


i) 
Nien 
iit ) , 
iy i Hon 
[b 
yo 


7 | 
*Yerameny 


a lication. 


DPA CONSULTING LTD, sb 


rchive Copy Sie 
Not to be removed A 


Pro; 
ject No, ine Vem 


me 


AN 
EVALUATION ASSESSMENT 
OP HIs 
SERVICE BUREAUX PILOT PROGRAM 


iy 


PREPARED FOR: PREPARED BY: 
The Task Force on DPA Consulting Ltd. 
Service to the Public October 1981 


Supply and Services Canada 


@ : ice ne vy 
=< na my « a j 


a 
a a) J a > > * 
Oe pe a 
pra Lead ’ \ y 


TABLE OF CONTENTS 


SECTION 


Executive Summary 


Introduction 


Service Bureaux Program 
Component Profile 


2.1 Mandate and Objectives 

220 Description 
232418 “Hrstory 
2.2-2 Service Bureaux Pilot Project 
Dic L eS Delivery of the Project 


2.3 Resources 


Models of the Pilot’ Project 
3.1 Program Logic Model 
3.2 Program Context 


5.3 Program Hypothesis Model 
Preliminary Assessment Findings 


Evaluation Issues 

5.4 “introduction 

5.2 Design Issues 

5.3 Service Level Issues 

5.4 Program/Related Dynamics Issues 


5.5 Evaluation Issues Summary 


valuation Options 

Rationale for Evaluation Options 
An Ex-Post Evaluation Option 

An Effectiveness Monitoring Option 


A Management Monitoring Option 


Criteria for Assessing Options 


a 

: 

2 

3 

4 
.5 A Formative Evaluation Option 
6 

i 

8 Recommended Evaluation Option 
9 


Evaluation Study Terms of Reference 


An Evaluation for Experimental Projects 


PAGE 


aes 
De 
22 
28 
33 
oo 


4} 
ou 
42 
43 
44 
45 
46 
48 
20 
Bid 


Digitized by the Internet Archive 
in 2023 with funding trom 
University of Toronto 


https://archive.org/details/31/761117098855 


EXECUTIVE SUMMARY 
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The focus of the Service Bureaux pilot program is PIV MLS2) 
to test alternative means of enquiry Service in 

improving the Canadian public's access to its 

federal government. The Service Bureaux role 

is to provide a supplementary linkage mechanism 

between the public and the federal government to 

facilitate improved access to federal programs 


ana@d services: 


As a condition of the funding .commitment to the Task 
Force, ,Preasury Board stipulated a,rtequirement for 
an evaluation of the Service Bureaux pilot program 
to be conducted prior to further funding. While 
funding has since been extended to October 1982, 

the purpose in undertaking an Evaluation Assessment 
at this time was to present management of the pilot 
experiment a selected range of evaluation issues, 
approaches and options for consideration as an aid 
to providing the best experimental results possible. 
The Task Force retained DPA Consulting Limited to 
conduct this study. 


During the course of the assessment, a profile ie Sam (es gots oD Mi 
of the Service Bureaux pilot program was Appendix. '"B" 
developed. Included in this profile were reviews 


of the mandate for:this.initiative, the objectives 


of the pilot experiment, the experience to. date, 
and as well, a preliminary assessment of the 
program s perrarhance 2 "Three (3) *modulesor 


the Service Bureaux pilot were developed: 


—- Program Logic Model; Biaths Oka 
- Program Context Model; 322 (15616) 
- Program Hypotheses Model 3.3 (27-29) 
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Interviews were limited to Task Force personnel 4.0 (19-21); 
at management's request. The information APPENDIX 'D' 
obtained in these interviews served two purposes. 
The first purpose was to provide the basis for a 
preliminary assessment of program performance to 
date.) Our findings “included: 
- There appears to be considerable concensus 
that promoting and facilitating the Canadian 
public's access to the Federal Government 


is a desirable objective. 


—- There 12s a néed ‘to, clarity. the role of ‘the 
Canada Service Bureaux in several policy and 


activity areas. 


- Freld resources find that the absence of more 
Gefinitive directions from and communication 


with the Task Force is a concern. 
- The public is not aware of the Service Bureaux 
pilot program. 


—- Advertising/promotion must be an integral element 
of the experimental approach adopted in the pilot 


project, 


The other purpose of the interviews was to 


focus on evaluation issues. 


Sa 
e 
(=) 


Fourteen (14): evaluation issues were identified ( 22-40) 
(see: Opposite page). These issues cover the 

range of evaluation considerations under the 

OCG Guide on the Program Evaluation Function. 

For purposes of management convenience the issues 

were grouped under three(3) program issue headings: 

Program Design, Service Level and Program/Related 


Dynamics. 
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Resounces, for ‘the pilot, progect, are‘ provided. to: the 
Task Force by Cabinet decision and Supply and Services 
Canada based on Treasury Board submissions. Asa 
condition of the original funding commitment for the 
service Bureaux pilot project, Cabinet required the 
Task\Force to report on the status of the project by 
January L902. "In ‘recognition’ Of “this reporting 
requirement, management of the Task Force decided to 
conduct an Evaluation Assessment of the activities of 


the pilot project, aS a first step toward evaluation. 


Proposals were called by the Assistant Director, Access, 
of the Task Force on Service to the Public and DPA 
Consulting Limited was retained to conduct the Evaluation 
Assessment... The purpose of this report is to present 

the findings and recommendations resulting from the 


Evaluation: Assessment conducted. 


1.2, Purpese and, Scope 


The purpose of this study was to conduct an Evaluation 
Assessment of the Service Bureaux pilot project and to 
Present £o management Of the Task Force on Service to 
the Public a selected range of evaluation issues and 
evaluation options for consideration. In accordance 
with the guidelines established by the Office of the 
Comptroller General on the program evaluation function, 


the study was designed to provide the following products: 


. A program component profile including the legal basis, 
ACtLIVEELeS, \products and ‘structure Of therprogram 
component, showing the causal linkages within the 


program; 


- A model of the program components covering: 


Objectives, activites, products and general structure; 
- A list of relevant evaluation issues; 


-. A number of possible evaluation options including a 


description of the techniques to be used and the estimated 


resources and time requirements for each option; 
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- Criteria to be used by Task Force management to 


determine the most appropriate evaluation option; 


- A recommendation on the option considered most 
appropriate by the evaluation team, including the 


rationale for the recommended option; and, 


- Broad terms of reference based on the selected option 
to assist Task Force management in the conduct of the 


subsequent evaluation. 


Under. the, reporting: Constraints; of) the! Task Force, 

the study was initiated in late August 1981 and targeted 
for completion by the end of the fourth week in September. 
Fact finding in the Evaluation Assessment was limited to 


members of the Task Force, both in Head Office and in the field. 


1.3 Approach 


The study was divided into four major stages (Appendix 
B: Study Work Plan): 


Stage, 2: Development of Program Component 


Profile and Context Model (s) 


Stage If: Assessment of Program Structure 


and Identification of Evaluation 


Pesues 
Shere be (een ESV ae Development of Evaluation Options 
otagecs ive Development of Terms of Reference 


A steering committee composed of the Assistant Director, 
Access and the Manager, Service Bureaux as well as a 
representative from, the Office.of the Comptroiler General 
was established to monitor and to advise the project team 


as. the study, progressed. 
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At the end of each stage, the study team met with the 
steering committee to review progress and to 
secure key decisions (Appendix C: Minutes of Steering 


Committee Meetings). 


A significant feature of the study approach involved 
extensive consultation with Task Force members and 
interviews at the field level. A standardized interview 
guide was developed covering the potential major evaluation 
issue areaS as well as, the details of processes, resources 
and utilization. In addition, a small but representative 
sample of both Itinerant Officer Mobile operations and 


Service Bureaux facitities, were visited. 


In the course of the study the consultants interviewed: 


- 6 representatives of the Task Force on Service to 


the Public’ in Ottawa; and 


- 17 representatives of the field operations of the 


Service Bureaux pilot program. 


Appendix D provides a list of ianterview-.contacts and the 


general format of the interview guide questions. 


A number of pertinent documents, reports, correspondence 
and prior studies were reviewed by the consultants during 
the study. These are enumerated in Appendix Es List of 


Source Documents Reviewed. 


At the conclusion of Stage LII of the Work Plan, a range 
of evaluation options were presented to the Task Force. 
THis »4ncterim report is included “as Appendix FF: Interim 


Report on Evaluation [ssues and Options. 
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2.0 TASK FORCE/SERVICE BUREAUX PROGRAM 
COMPONENT PROFILE 


2.1 Mandate and Objectives 


The Task Force mandate for the Service Bureaux Pilot 
Project is found in the Task Force on Service to the 


Public Discussion Paper (July, 1980): -- 


- To work with all departments in raising the 


quality of government service; 


. To improve communications between the public 


and the public-service; and, 


- To make it easier for people to deal with the 


government. 


The mandate presented in the Discussion Paper 


embraces three (3) major activities: -- 


- Access to government (e.g. by telephone, through 
in-person visits, through Members of Parliament and 


production and distribution of an Index to Programs and Services) | 
- Awareness of the Federal government role; and, 


- The supply and delivery of service and information. 


The Minister of Supply and Services Canada reports 
to Cabinet periodically on the status and activities of 


the Task Force. 


The objectives Of the Service Bureaux Pilot Projece 
are also. stated in the Task Force on Service to’ the 
Public “Biscussion Paper (July, 1900) 2." fixree, Che 
pilot project 1s to assist the Task) Porceyinimee ting 
the Cabinet requirement "to determine the level of 
service, the cost effectiveness and market penetration 


levels required to meet the needs of Canadians."; second, 
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the pilot project is intended "to find ways and means 
of attacking the problem and to develop an approach 
to jsol ving sb." : 


2.2) Description 
22) History 


The Task Force Service Bureaux pilot project is not 
au isolated event, but rather one of a series of 
initiatives that evidence the Government's continuing 
concern with the ways it communicates with the Canadian 
public. The failure to communicate effectively was 
first noted by the Royal Commission on Government 
Organi zatiorm (1962). -in*1969: ‘the’ Task ‘Force “on 
Government Information 'advocated that the Government 
create an entirely new kind of central agency, to be 
Called Information Canada, around which a number of 
existing and potential information functions 

could cluster and through which the Government could 
focus ats efforts to improve communications with the 


public. 


Six years later, Information Canada was dismantled 

after an unsatisfactory experience. In 1977, Cabinet 
again sought a comprehensive analysis of the state 

of the art. The Treasury Board Task Force was established 
‘to examine the nature of the relationship between the 


federal public service and the public it serves. 


The studies undertaken by the Task Force indicated that: 


. the public perceives major difficulties in gaining 
access to the Federal Government to obtain services 


and information; 


. there is a serious lack of awareness of federal 


programs and services; 
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. tnere- Ls" uncertainty aboutithe actualerole of the 


Federal Government in the community; and, 


. the public views the Federal Government and the 


DUDIic Service as indifferent, hostile and ansensitive. 


As a result of these findings and a general dissatis- 
faction with the image of the federal government as 

well as‘ithe’ publ le*sService,* ands tos ensure ‘that»corrective 
action was taken, the current Task Force was established 
In? tate 19.78% 


The DSS Task Force on Service to the Public identified 
three -(3) target areas for action: Access, Awareness 
and Service Delivery. -»The Awareness and Service 
Delivery components are in their formative stages. 

The Task Force has focused considerable attention 
however on attacking the problems that Canadians 

have in accessing and obtaining Federal Services and 


anftormation. 


Many respondents to earlier studies have indicated 

a preference in dealing with government on a 

face-to-face basis. Personal contact is considered 

better than the telephone when discussing the completion 

of forms, looking up pamphlets or discussing correspondence 
which may be causing problems. The Service Bureaux 


Pilot Project is designedirtoitest this; public meed. 


2.2.2 Service Bureaws Pwlot IrerTect 


The pilot project is)!comprased@ oficfour (4)* modules 
including Service Bureaux (8), Satellite Bureaux (6), 
Ttinerant Offices (8) and Information Carrells (30). 
Each of these points of Service have been placed in 
communities across Canada so as to maximize the range 
of economic and social strata involvement in the pilot 


project. 


12 any 
fame bit Hoe varie aT Tha! 


7 
r 


ais retinas ane ‘ietenle 


IVES PON 4, J wih rite per: itl 3 


PB oy 
rwarteep ) (fk 4aW RS. Haet JaaeIUD: ined Peake 


ee ol f ma i 
: ~ wk, 


an ‘ | aN 
; f i" ‘ rR 9k hey : Dial 7 oe 
os Bh 94 cae Pe Se at re, poet fe iat Dt a eat wi 
; = 7 | ; A . ; ig ky 
STSWA: ROOK. «| NOMEN: Mk eee eas ieaiaee iey kod rah 
i : high ate : ; at 
p F ss - rt 1" 
Th ESSMISTAWA BP oy ie 
err 
‘REDN I S9 SVE Sea ee See are tac, vo D hae’ on 
ie Fae | Def gt BE i Di eet ss Na: daeee et THEY 
er eek ped 2h Bie? Ly Ay ig ri i ye hens 8) Be no edmond . 
Yran ie. Shee pripetereen Bie werk segaun a See . 
if at CLD La een hey s 
. a ae 
; bd ' 


Mig, 2S Divisd Gora ses, ER ie ekahel sie SC eed 64 


o> Tere tog SW OT ee. eo eee ene es 


i tap ine jy ryysceiroe — | hie sean 4B@bend: aant Sel > byt? 
“ its: . i 
. it ry comer eeiottiqe fon ait eset Se gaget 
! mer 2a pea kienier Cie fe DSM), SR By 
' a. : : - Ww 


BLDC IE, “UES LE a> 3M" SCALMOTS rh asad ee pane. iii i ciw 


Muy 2 sae ea ganiptand ec auf +L RY 


re 


peoca 9 ne jbl 


iy 
wan, Tb) Spee osit est Bk ‘posts 
) rr a al Ga mike a 


ty ‘ . weit re i own li 


Re) 


Each module of the pilot project is intended to provide 
enquirers with prompt, accurate referrals to Federal 


departments/agencies. 


2.2.5 Delivery Of “the Pro,ect 

The Task Force on Service to the Public received Cabinet 
approval of the Service Bureaux proposal in October, 
1980. The First Service Bureaux was implemented in 
Edmonton in December, 1980. Accordingly; the ‘Pilot 
Project is in, only the’most formative of-stages: Since 
thas Cime, the Task Force has focused attention on 
mounting a credible field project of Bureaux, “TFtinerant 
Offices and Carrells ‘within’ very strict time constraints. 
Thirty Telidon-equipped carrells will be implemented in 
October 1981, placement of an additional sixty non-Teladon 
Garrelis 4s "planned for the 1982-83 fiscal year. Exhibit 


2 presents the Service modules and effective implementation 


dates. 


The Manager, Service Bureaux provides national policy, 
administration and coordination of the Service Bureaux 


pilot project. Regional Managers are responsible for: -- 


. delivery of service; 
- regional data bank development; 


.- regional liaison with federal 
departments/agencies and other 
imrormatronm Services: 


~4regional liaison with other tederal 
departments/agencies e.g. DSS/ 
supplies, PWC/accommodations; 


. facilities management, administration 
and immediate personnel supervision; 
and, 


. aS appropriate, supervision of satellite 
bureaux and itinerant services. 


a) Os . Le 


tarcin® bi igi about ai) ae > 


 ~tiwdoleng) pi iemegpon sip “y | nde at 
re or TRIN f atthe Rew iueorua aoherie desis Aun 
$OLL9 ais Ebert Brags OBE eT. gto aXh ‘ak im i 
Sou 08 2a. le A0avil {set Se sie edt its. ey et dostoit. 
12 i trays bead HOS: Eri S4aR ARG. es » pial oeyt? 
POSTION RUMI ES Soeyy ey. Ef toi <p Ley 2 a iit 9 sence 


TONE rey 


E29 AY taney obras) Fee tes: Cay, | ‘us dys eh cme ie Le nti ie bate 
( Sageray pies el eo SCE Oe Pag cle “Hie Gio we Thiet 
woe 1! yy oct? al easzet aia See st ena, '‘; wfaet sched Se 
7 i } ee a: : CPi: cul } 4 j sits f 4 ee Oe ea 
t " P 7 7 
cA 2 ry : ' iy yr a fil as vo} Pawaheyeg § 
, see 
4 ; ‘ afi ; od a | Tie 
; 4 I ! \ . 71 
ij i. {) d H ‘Shee: > ee ns || 
it , ] 5 ’ Li j mf ty ‘ f* ‘c t; } “J 
hh) Vv ; ; 
a i i 
‘ ‘ ‘ i bare j i ee a ee t i 
» Cotte ie 6). Peneeeivrads <, 
{? i } ead us . 5 ry , (An =i ye A ie eo ie | tye f 
. “aii a TES mes; 5 
| 
Ty « ry 


1 <i yatta mit thigteat: : 
al pedi ie ans aged Wisyob. 
Ligeyt Fb ver re) a wi 
tab ha a i , tremoeighetlenn Adio. i . 

got | Saeed wat Somes we ; 


ee aie 7 
aka on nuevas Sry 7 
ve aoe : 


aD rf 


H 4S a in ae | joel 4 i) j ' i ee Lime 


Fi 


rears 


| 1 ne 
. ee : oak ane 


ter 


ek ties : . i, _ 


ec ee eee cee eee ee ee eee eae ee eee ee 


BAHIBIT. 2: PILOT: PROJECT SERVICE BUREAUX ee AND EFFECTIVE 


IMPLEMENTATION DATES 


Service Bureaux Description/Observed Effective 
Module/Location PYtTanie te. Date 
Service Bureaux 

Vancouver Commercial Core ST O37 oO i 
Edmonton Active pedestrian 9/12/80 
Regina Heritage, Core 207077 SL 
Winnipeg Core shopping 15/07 7381 
Toronto Core office/shopping 15709/81 
Quebec Binie Collar arscrice TFO57 8h 
Haljtar Storerront, core 1/04/81 
St. goeun s Shopping,mall T/O0C/ SL 
Satellite eas cytes 

Victoria Federal Building 1/08/81 
Calgary Federal Building 15/07/81 
Saskatoon Federal Building 1/7 087,58 
Montreal Core of fice/shopping Z27O0GI SL 
Moncton Federal Building 20/077 81 
Charlottetown Federal Building : 20/07/81 
Itinerant Services 

Kelowna Enter1or yy Bec. Ley aU ney ars We 
Gimli/Interlake Rural/Industrial 1/08/81 
North Bay Northern Ontario 2O/OTLSL 
Sherbrooke Eastern Townships 20/07/81 
Matane Gaspe Peninsula 277 ONS Od 
Sydney Cape Breton 6/0778) 
Bathurst Hinterland 25/7 097-31. 
Windsor Central, Nfld: L5/ 08/81 
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Exhibit 3 presents an organizational overview of 
the regional operations of the Service Bureaux pilot 


project. 


Consistent with the experimental nature of the pilot 
project, the Task Force has created Service Bureaux 

in a variety of locations -- from major urban core 
shopping concourses to street level/store front accommo- 
dation in commercial/financial core districts -- to test 
possible key locational variables against the market 
determinants of demand, penetration, capacity and cost 
erficiency/etfectiveness. Similarly, the pilot project 
Ltanerant Services cover several types of smaller 
communities of 10,000-30,000 population in regular, 
scheduled tours. In the Interlake Region of Manitoba 
however, a unique dynamic is being tested as the 
Itinerants serve small communities of 1,000-3,000 ina 


mixed rural/light-industry Setting. 


The Vancouver Service Bureau is staffed by federal 
public servants; in all other cases, regional personnel 


are private sector employees. 


Detailed records are maintained on each enquiry trans- 
action and a statistical base is currently being 
developed by Regional Managers. The summary data 
available to the study team during the formative stages 
of the pilot project indicatés jarsignificant potential 
demand for an in-person federal enquiry/referral service. 
Data displayed in Exhibit 4 is from the Edmonton 


Service Bureau. 
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EXHIBIT 3: ORGANIZATIONAL OVERVIEW OF SERVICE BUREAUX 

PILOT PROJECT REGIONAL OPERATIONS 


Sess 


Manager / 
Service Bureaux 


Regional Mgr. Interlake / 
Winnipeg / __Gimli Region 
Manitoba Ttinerants 


Regional Mgr. 
Regina / 
Saskatchewan 


a Satellite 7 
Saskatoon 


Regional Mgr-~ 
Edmonton / 
Alberta 


Satellite 
Calgary 


Regional Mgr. 
Vancouver / 
British Columbia 


Kelowna 


Ttinerants 


Satellite 
Victoria 


North Bay Regional Mgr. 
Leynerants = Teronto/ 
Ontario 


Satellite 
Montreal 


Regional Mgr. 
Quebec / 
Quebec 


\. Sherbrooke 
Ttinerants 


Matane 
Ltinerants 


Regional Mgr. 
Gr yon Se 
Newfoundland 


Windsor 
Ltinerants 


Regional Mgr. 
Hatitas / 
Maritimes 


Satellrte i 
Moncton 


Sydney 
Ltinerants 


Bathurst 
Itinerants 
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EXHIBIT 4 — EDMONTON OFFICE OPERATIONAL DATA 


Operating Volume of % Change In Number of Enquiries/ 
Month Enquiries Enquiry Volumes Staff Staff Day 


Of Previous Mo. 


249 . i 


1 2 S89 
2 353 AN PMR 8: 2 8.8 
3 388 9.9 3 529 
4 265 C3 7) 8 4.4 
5 302 14.0 6 yet 
6 ew AAS 3 6.6 
7 384 CLUDE P) 4 AL 4 
8 300 (21.79) 4 Shaye 

Mean (Standard Bo9. (66) 64 f 30.0) - 5.6: 41263) 

Deviation) 

Periodic Growth 2 AS 

Rate 


The Gdentification, advertising, and promotion of thé Service 
Bureaux/Itinerants is an integral part of the Vask Force 
pilot project. A nattonal promotional strategy as currently 
being prepared by the Task Force. However until implemented, 
the promotion of the Service modules is restricted to 


official opening ceremonies and local signage. 


pees Resources 


The Service Bureaux pilot project is allocated Task Force 
resources through submissions to the Treasury Board and the 
Estimates of Supply and Services Canada. The resources 
include: person years, salaries, operating or non-salary 


expenses and professional services. 


Pe aed. Bes I 


‘a 

a ‘oe 
eae Sear rats 
of n: ee otis z s 


%. 
rprs 


va Pers : 
a. 
Le» & he . 
im ae - P 
7% a rr rs 
oe a 
% 


© e* 
he » 
: 
’ 
i 


of 
= r 
hes ? ” 
{ i] « 
a) 
yi 
4 1 
: ; pa 
Pear & } 
Ps oct 
- - rd 
ace | Pa2r¢ . ee | Me RL 7 
Ls Mévg Pe Ret) Se A #21 <a yy Vee ORF 


oP ood ‘apokmpime sal et re 
il | . 


«SBIR | 2 + aires. a Bday 


ww 


The Task Force is integrating its internal financial manage- 
ment operations with the Department's resource cycle and 

for the fiscal operating year 1982/83 the Task Force on 
Service to the Public will be included in the Department's 


Main Estimates Submission. 


In the course of the Evaluation Assessment, available summary 
resource data were examined by the study team. Detailed 

financial statements for the Task Force Service Bureaux pilot 
project, Service Bureaux modules, Service Regions and Offices 
were not available at the time of the study, however work was 


underway to develop these reporting formats. 


It was not possible within the study team's time frame to 
develop a detailed description of how management of the 
pilot project plans and controls expenditures and budgets 
from existing reports and documentation. Detailed DSS 
requisition statements were available, however these 


must be reclassified to provide the financial data required. 
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a. MODELS OF THE PROGRAM 
Ral» Program Logic Model 


The Program Logic Model, Exhibit 5, was amended after 
consultation with Task Force management. In its amended 
form the Model received general approval from field 


personnel. 


Three srgmificant points emerge” from an analysis of the model: 


. the General and Specific Outputs have been verified 
anorevidenceiwas found that, within the variations: of 
approach permitted, each Bureau and Itinerant operation 


provided the output described; 


. although tne Immediate, Intermediate and Long Term 
Impacts and Effects were those intended by the 
Program, little evidence exists of the ability of 
the Task Porce “to measure the levels of effects or 
their application to a continuing program of access 


to Federal services; and 


. Unintended impacts have not been reflected as the 
depth of the Assessment study did not permit access 
to affected groups, but an example might be the 


reactions from M.P.'s and their constituency office 


Starts. 


The linkages of Objectives, Activities and Products to the 
Impacts and Effects Intended appear plausible. At the 
Intermediate and Immediate levels particularily, there was 
superficial evidence of objectives achievement in the provision 
of services. The evidence of achievement in providing pertinent 
data for analysis of the Program for future design was less 


aprarent: 
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Sine Program Context Model 


The basic premise, displayed in Exhibit 6, is that the 


Canadian public requires a supplementary bridge to obtain 
access to Federal Services. 


The use of the phrases "Informed Public" and "Uninformed 
Public" must be examined in the context of each transaction 
encountered. The same person could, under varying circum- 
stances, be “informed” in®Yone) situation and “Uninformed” 


in eanocrer. 


It YS the view of the gaudy team that the Program rationale 
is anchored on the ’bridging” premise. The current services 
being provided are to determine an optimum vehicle rather 
than to determine the public need for assistance. This is 


amplified in the Evaluation Options presented later. 
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PROGRAM CONTEXT MODEL (TASK FORCE ON SERVICE TO THE PUBLIC) 


Causal Hypothesis 1. 


There are segments 
within the Canadian 
public that have needs 
for information and 
assistance in 
obtaining government 
services. 


Task Force 
Service 
Bureaux 


Public needs for 
Information about/ 
access to/and service 
from Federal and 
other levels of 
government 


INFORMED” PUBLIC 


Causal Hypothesis 2. 


A supplementary linkage 
mechanism between the 
public and government 
services is a necessary 
means to facilitate an 
improved government 
response to public needs 
for service 


Federal 
and other levels 
of government 

services 


Members of the public may belong to either of the channels 
that are noted above in the flow of enquiries concerning 


government programs and services. 


Which channel the 


individual follows will depend on the specific issue at 


Be pariacular point in time, 
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343 Program Hypothesis Model 


The hypothesis model, presented in Exhibit 7, has been 
modified to reflect) the bottom line’ requirement: for idata, 
upon which to determine the level of service and optimum 


delivery program for the future. 


The use of a hypothesis structure is to provide a setting 
for the Program and upon which evaluation testing can be 


undertaken. 
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4.0 PRELIMINARY ASSESSMENT FINDINGS 


As indicated previously, in the course of the Evaluation 
Assessment, the. study, team conducted, a number of. formal 
interviews with individual members of the Task Force 

as well as a select sample of regional personnel in six (6) 
locations. These interviews were conducted primarily to 
assist in developing reasonable models of the pilot project, 
in describing its operation and developing Evaluation Issues. 
An initial sounding of the pilot project's performance, 
impacts and effects was also obtained. ‘The data gathered 
from interviews during the Evaluation Assessment do not 
represent statistically valid or broad-based samples from 
which to draw specific conclusions about the actual or 


anticipated performance of the pilot project. 


It was recognized in the study work plan that some preliminary 
assessment of the project would be useful to Task Force 
management. The following observations, from the comments 
collected through the interview and data gathering stages of 
the Evaluation Assessment attempt to provide insights. 


The key observations can be summarized as follows: 


. There appears to be considerable concensus that promoting 
and facilitating the Canadian public's access to the 
Federal Government is a desirable objective. Sample 


comments received included: 
- "...(we) are contributing to the 
perception of a visible and approachable 
Federal Government...” 


a". 2 (1it!as) essential’ to dispel the. ‘big 
they' of the Federal Government by 
emphasizing people and direct contact...' 
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- While there is agreement with this thrust as an 
objective statement, there is a need to clarify 
the role of the Canada Service Bureaux in a number 
of areas. Interviews with Task Force personnel 


identified the following areas of concern: 


- referral role versus the information role; 
—- publications role as. an aid to referral: 

- liaison with Departments and Agencies; 

- liaison with other levels of government; 


- liaison with other community and Service 
groups in the information area; 


- assistance to the handicapped/disadvantaged; 


= the definition of the target population(s) 
Vis avis the locational variables of Service 
Bureaux offices; 


- promoting the services of the Bureaux modules 
to the target population(s) through the media; 


- rank ordering of client groups according to 
Task Force priorities; 


- the need to clarify the interaction with 
Members Of Parliament and constituency offices; 
and, 


- criteria in the development of data bank resources, 
updating and central information available for 
this updating, 


In summary, an interview contact suggested that the -- 


-.-.-concept of the Service Bureaux program has changed 


from a narrow referral service to providing more service...” 


. While the experimental design of the pilot project necessi- 
tates flexibility and the Task Force has attempted to 
address many of the broader issues above, field resources 
find that the absence of more definitive direction from 
and communication with the Task Force results in a policy 


vacuum with each Bureau formulating ad hoc approaches. 
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. The feeling was expressed that the Canadian public 
generally is not aware of the assistance that the 
Service Bureau modules provide; is not certain of the 
meaning of a referral service; but those making 


enquiries were most appreciative of the Service rendered. 


- Based on the modest number of interviews conducted 
during the Evaluation Assessment, the majority of 
respondents indicated that advertising/promotion of 
the Service Bureau modules should be an integral element 
of the experimental approach adopted in the pilot 


project and should be Given oreater priority. 


The points highlighted in this Section are by no means all 
embracing. Lhe paucity of sound conclusive evidence, in 
the judgement of the study team, reinforces the need for 
further evaluation at this time, despite the relative 


newness of the Service Bureaux Program. 


These observations, in conjunction with the program models, 
were used to identify and assess a short list of evaluation 
issues of potential significance to senior management. 


These tare) enumerated in the next section. The evaluation 


issues identified reflect an attempt to be comprehensive 

in scope while sensitive to the experimental nature of the 

pilot project and its formative development. In addition, 

the overriding evaluation concerns ensure that the pilot project's 
processes, mechanisms and framework will permit the answering 

of management's questions related to the operational 


assessment of the independent program initiatives. 
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5.0 EVALUATION ISSUES 


el a THCLOCUCTCLON 


Within the current guidelines and approach to evaluation 
issues, as supported by the Office of the Comptroller 
General, four main issue areas are highlighted in this 


section: 


Program Rationale; 


Objectives Achievement; 
in MOpACES, and, Bitecte. and, 


- Alternative: Program Delivery. 


The evaluation issues identified are presented in Exhibit 


S& anad,Sover all four evaluation areas of concern. 


The issues presented in this section are grouped under 
three headings -- Design Issues, Service Level Issues, 
and Program/Related Dynamics Issues. The rationale for 
the issues and the expected results from their evaluation 
should be considered carefully before including or 
excluding them in the evaluation options. The evaluation 
methodologies to address various combinations of these 


issues are discussed in Section 6: Evaluation Options. 


5.2) Design Issues 


The principal objective of the Task Force Service Bureaux 
pilot project is "to detemine the level of service, 

the cost effectiveness and market penetration levels 
required to meet the needs of Canadians." The Task 

Force designed the pilot project, therefore, to enable 

the operation of and assessment of alternative service 


approaches. 
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EXHIBIT 8: EVALUATION ISSUES RAISED ACCORDING TO OCG CRITERIA 


S.C. Reet  -_—PO—Omr—x—Xvxmwrle— —————————— ee eeSeSSSSSSFFeFSssSs 


CATEGORY 


Evaluation Issue Raised A B 


Program Mesign Issues 


1. To what extent have Service Bureaux Program criteria been Xx X 
developed, adhered to and with what current results for 
the ,uture determinatton of operating modes and levels? 


2. To what extent has the Service Bureaux Program assisted x 
tn tne determinatton of the market to be served and the 
actual levels of demand to be met? 


3. das she development of data banks at local service x 
Levels contrtbuted to the tdentiftcatton of the 
publte needs for referral and/or access? 


4. Are the current tnterpretattons and practices followed xX 
bu the Servtce Bureaux Program adequate to assess the 
necds of Canadtans? 


Service “Level Issues 


5. Is tncre evidence that the Service Bureauz Program has xX 
resulted in uneremental cenranges tn the volume of 

anqurrries and changes tn cltent sattsfaction that would 

WALCETE AECOME Petes need for, or moditftcattcn of 

"am «services? 


6. Are the dynamzes of tne Service Bureaux initiatives xX 
supportive of eltents’® needs: 
= 22 DALK- En O77 tces? : 
—(|t ticnemane services: and 


= 2n @Garrelts? 


j. £oO wnat wacent hes the proviston of publicattons been 
a@ factor tn providing tmproved services to the publtie? 


8. Is the current limitatton of the Service Bureaux Program X 
role (e.g. referrals only vs. referrals and tnformatton / 
in-person pvrtortty vs. tn-person and telephone enqutry) 
providing a sujfietent level of service at the testing / 
experiment stage? 


&, 
. 


Yas the level of adverttsing had an tmpact on the Service X 
Bureaux Program operattons and to what effect? 


Program/Related Dynamics Issues 


~ »,4 
DOO wNaGe eLrerte ane tONwnat erpecte have 
- Federal Lepartments and Agencies, 
- Provincial/Muntcipal jurisdictions, and 
- re sates Members of Parltament and thetr 
Constttrenecy Offtces 
played a role in the Ptlot’ Project tnttiatives? 
il, ow nave internal management, tratnitng and communtcattons 
affeetcd tne operation of, and the achtevement of, the 
Service Bureaux Program projects, both regtonally and 
CPC OLE. 
12. Is the’ ltevet of and use of resourcing avatlable and 
sufftetent to centeve the stated objecttves of the Program? 
13. Snculd a Pregram of providing access / referral to the X 
sublte be manned and operated through Publte Service 
etacr or trroucn continutng rrivate contract cavrrange- 
ences ane what are the Gunarites, oeneytts, ana GrawpaeKks 
OF each azvroacn? 
: Do the office locations, layouts, image portrayal and x 
ssaPe Gynamtes in nandling the public, nave a bearing 
9n she volume and tupes of pubdlie use and to what effect? 


A Rationale CC limpacts/Eprects 


B Objectives- Achievement D Program Alternatives 
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The Service Bureaux modules were to be designed, located 
and operated, not so much to provide a pre-determined 
level of service, but) to experiment. with a, range of 
diteerent Senvaces< (The purpose in: this-approach) was to 
determine appropriate mixes of locations, levels and types 
of services for future decisions to be made based on their 
findings. In order to achieve this objective, the design 
of the Program must -- of necessity -- be predicated 

upon Clear criteria and carefully constructed recording 
and reporting mechanisms as the basis for the future 


assessment Of thes pwloti project: 


Several Canada Service Bureaux and Itinerant facilities 
were observed. Office locations were varied but a 
Significant question to be evaluated is whether Program 
criteria are suitable for providing answers for determining 


the ultimate delivery model. 


In the area of services provided, the interviews and 
field visits indicated a variety of service approaches. 
The value of each different type of approach should be 
the provision of comparative results. Whether the 
means to assess service approaches exists and to what 
extent it is meaningful would form a key part of a 


subsequent evaluation. 


All Bureaux and Itinerant offices visited maintained 
records on the numbers and types of enquirers and their 
enquiries. No clear evidence was found on whether the 
information forwarded to the Task Force was used. The 
basic premise of operating the Service Bureaux Program is 
to utilize such findings and therefore the inclusion of 


this aspect of the Program in an evaluation is evident. 
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The issues to be resolved within the program design 


aise aS cur Crs 


Issue #1 To what extent have Service Bureaux Program 
ertterta been developed, adhered to and wtth 
what eurrent results for the future 


determtnatton of operating modes and levels? 


Because of the importance attached to the results of the 
Pilot Project in determining and shaping a continuing 
program of Service to the public, the foundation upon which 
judgements will be made is vital. Criteria, designed and 
applied vigorously are the only verifiable measures in 
answering such questions. Le is therefore essential that 
the Task Force management obtain a clear indication as to 
the edequacy of existing criteria, their application; and, 


indications if new of modified criteria are required. 


The issue is of special Significance in view of the growth 
and development of a range of other information/service 
groups advising the Canadian public on access to the 

public’ sector. Should the criteria and structure of the 
Service Bureaux pilot project reflect these concerns? 
Should the criteria recognize the potential. for greater 
liaison with the services of departments/agencies and other 
levels of government? Until the criteria and their impacts 
are analyzed and evaluated these types of questions cannot 


be answered with certainty. 
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Issue #2 To what extent has the Service Bureaux Program 
assistedsin the deterntnatton,.of the market. to 
be served and the actual tevels of demand to be 


met? 


Despite the previous experience of Information Canada and 
earlier Task Force research, the question of where the 
greatest needs exist and how to reach these publics remains 
largely ‘unanswered. The use of Bureaux and Itinerant 
services in varying locations and modes was intended to 
assist in this market identification, definition and analysis. 


Early evaluation results should indicate the level of 


success possible through these initiatives. 


Interviews with Task Force management repeatedly raised 
this, assue. as an,area,of.iamportance.,,is there a need for 
the services being provided? How do we know what kinds of 
information meet customer needs? Has the public an aware- 
ness of the Program? If so, at what level? Is the Pilot 
Project providing a meaningful test of the dynamics 
Cabinet wished tested? Evaluation of this issue would 
provide the basis for determination of these concerns with 


some certainty. 


Issue: #3. Has. the development.of, databanks at, Local 
service levels contrtbuted to the tdenttfteatton 


of the public needs for referral andor cecess? 


The premise of the Service Bureaux pilot project is the 
appreciation of and sensitivity to the Canadian public's 
need for access to the federal government. Consistent 
with the overall experimental design, the Task Force did 
not provide Regional Managers with a complete/comprehensive 
data bank. Rather, Regional Managers were familiarized 
with the basic building blocks of the national data bank -- 
including, the 1981 Index to Programs and Services -- 

and were encouraged to build local data banks about this 


superstructure. 
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Interviews with Task Force management and Regional 
personnel indicated a wide variety of approaches in the 
development of the local data banks were adopted. Field 
visits by the study team confirmed the diversity of | 
approaches. Several personnel indicated that the strengths 
of this approach were evident in the sensitivity to local 
priorities, immediate information issues, increased staff 
involvement/morale and the high quality of referrals/ 


reported Ciient. Satisfaction. 


ine Mssue adentified here Ws’ of Significance therefore to 

Task Force management and Regional personnel. Evaluation 

of this issuve will have a direct bearing on the whole 

range of design variables antegral ‘to the concept Of» appreciation 
ofan sensitivity to the public's need for amproved access 


to government information and services. 


Issue #4 Are the current tnterpretattions and practices 
followed by the Service Bureaux Program adequate 


to assess the needs of Canadians? 


Over the formative period since the introduction of the 
Service Bureaux pilot project, the Task Force has delineated 
the roles of the Bureaux in a number of discrete policy 
decisions or management interpretations. A sample of these 
deer noLced : 
- to focus the service initiative on strict 
referrals to programs and services; 


- to expand the referral function with the provision 
of basic information as warranted by the enquirer; 


- to emphasize the in-person dimension of the 
Service Bureau modules; 


- to 'discourage' the excessive use of the Bureaux 
modules for responding to telephone enquiries; 


— to distinguish the in-person services of the 
Service Bureau pilot project from existing joint 
federal-provincial telephone enquiry services as 
well as other information and referral agencies 
in the community; and, 
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="to* limit ‘the profile of /the “Service: Bureau 
modules by reducing advertising/promotion 
activities and the employing of nominal 
Signage for identification purposes only. 


The evaluation issue posed is whether this approach, if 
adequate in the past, should be continued or should a more 


formal, directive approach be adopted. 
eg) Service Level Issues 


The major thrust of the Service Bureaux pilot project is to 
identify ways and means of shaping a Program to meet the 
public's needs. The effort to produce a viable level of 
Service 1S etco,a;siqnifacant parteol the pilot project 
experiment. Only through the best service efforts at the 
Pilot Project stage, can improvements or need to change be 


assessed in a comprehensive manner. 


The study team recognizes that it is therefore within the 
scope of the evaluation to examine issues of current 


effectiveness of operation. 


A number of evaluation issues therefore emerge from the 


level of service area: 


hocsues#5. Letherey, evidence, that.the, servurce Bureaux 
Program has resulted in.ineremental changes 
in the volume of enqutries and changes tn 
client, satisfaction, endt, would, tudizeate @ 
continutng need for, or modification of 


Program services? 
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Thesvery naturevor athe Bureaux,’ storefront" elocations will 
attract a clientele. More obviously the Itinerant services 
reach »peeple where,;they work, or reside in rural areas, saving 
the spublicstransportation and) comunication costs .ii'The issue, 
however, 1S whether these services have increased the number 
of public served or have they merely siphoned off a public 
who might otherwise have found the appropriate: Department 


providing service. The incremental effect, in our opinion, 


Po tne cruel test. 


Interviews with management emphasized the Task Force's 


awareness of this issue’ and their need for information: 


- "Is there a need for the service provided"? 
-"What is the best mix of services to be provided"? 
-~"How does client satisfaction rate"? 

-"What are the dynamics of service..."? 


—-"What are the implications for improved delivery 
of the Program's Services”? 


Issue #6 Are the dynamics of the Service Bureaux 


i’ctLtatives supportive of ettents’ needs: 


— wen walkatri of feces 
—4an ebinerant, senvicess rand 


— Li» CarreLlls 


Despite a common, broad-based training session with most of 
the field staff, the study team observed many different 
reception/assistance techniques in use. In some cases the 
training-recommended techniques were challenged. As this 
area of public interface (which in Departments is so often 
criticized by the public) is critical to the Service Bureaux 


concept, we believe it to be a significant evaluation issue. 


Evaluation of this issue would clearly isolate the activities 
ana the products of the pilot project ensuring that the 
intended publics are in fact reached. The measuring of the 


effectiveness of supportive client dynamics would provide management 
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with a factual data base of use in refining existing 
activities and in the formulation of future program 
initiatives. A further benefit derived from the evaluation 
of this issue would be a determination of the extent to 


which the target publics of the pilot project are reached. 


Issue #7 fowhat extent has the proviston of publttca- 
ttons been a factor in providing tmproved 


services t@ the pupttes 


The pilot project initiatives have been designed to provide 
a spectrum\or locations and activities so a means of 
providing information for the design and operation of a 


continuing access and referral service. 


The availability and suitability of a range of government 
publications is one aspect of the function of an access/ 
referral activity designed to inform and assist Canadians 


in obtaining service. 


The way in which publications are presented; the ‘type 
of public attracted by a publications display and the 
use to which publications are employed are all questions 


that should -be an, integral part,.of a fonmal.evaluation. 
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Issue #8 Is the current ttmttatton of the Service Bureaux 
Program Yoler(evg..eeferrats only vs. referratle 
and tnformatton/in-person prtority vs. tn-person 
and telephone enquiry.) providing a sufftctent 


level of service at the testing/experiment stage? 


There is an almost insatiable appetite for service and 
information yet, nobody ever says -- you have given me 

all the service I want. The Service Bureaux pilot 

project is a formative experimental design. It is intended 
to test a range of complex, integrated issues (e.g. 
locational variables, demand characteristics, advertising 


AINDACTS cet) 


The focus of the evaluation issue addressed here centres 
on the efficacy of in-person referral services in producing 
appropriate levels of service over the duration of the 
pilot experience. The importance of this issue is to 
identify and to test the legitimacy of the over all pilot 
project initiatives in view of the real policy constraints/ 
limits imposed on its operations. Key questions that the 
evaluation would examine include: 

, - Does the Service Bureaux pilot project in fact 


improve the access of citizens to government 
information sources? 


—- Do members of the general public experience 
reduced. frustration. in,obtaining services ;and 
information through the Service Bureaux network? 


- Is the image of the federal government as a 
provider of services” across” the land ‘enhanced 
by the Service: Bureaux .pilotaproject? 


—- Does the new enquiry system provide useful 
assistance to Members of Parliament and their 


staffs in meeting the needs of their constituents? 


~'Is the image of the federal civil servant 
enhanced by the Service Bureaux pilot project? 


=What are the implicearcions for level of service 
as the program matures? 


- What are the main service related problems 
experienced? 
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~'How tong will it take to resolve’ these 
service related problems? 


- Are the pilot project's basic service priorities 
consistent with an appropriate level of service 
at the testing/experimental stage? 


Findings in response to these questions would permit both 
the evaluators and Task Force management to make substantive 
statements on the efficacy of the specific range of pilot 


project services and the Service Bureaux concept. 


rssue #9O Has the tevel of-advertising had an tmpact: on 
the Service Bureaux Program operattons and to 


wnat effect? 


An extended promotion/advertising campaign highlighting the 
purpose, role and availability of the Bureaux, Itinerants 

and Carrells is awaiting Ministerial approval. At this 
point, the pilot project service modules have had the benefit 
of only limited local signage and office opening promotion. 
Interviews with Regional personnel indicated this, in itself, 
created a problem as the first questions a potential client 
sought were -- "What is this? What do you do? What is a 


referral? Who 2s responsible for this?” 


The effects of advertising/promotion of the Service Bureaux 
would be complex, interactive and time specific; the impacts 
due to the absence of a concerted promotion of the Service 
Bureau modules are equally complex. Evaluation of the impact 
Of “advertising on the pilot “project would ‘focts *on ‘specific 


client awareness and demand determinants, including: 


—availabilaty of infomation explaining he 
program; 


. understanding of overall program objectives; 


. awareness of the location of the nearest 
Service Bureaux module; 


. speed and accuracy of response to enquiries; 


. robe. and participation of the federal depart 
ments in responding to complex questions 
requiring interpretive ability. 
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Evaluation of this issue would be essential in the determina-— 
tion of the importance of advertising and the service demand 


related impacts in an experimental program of this type. 
5.4 Program/Related Dynamics Issues 


This section presents the study team's analysis of the 
various dynamics that could be expected to have significant 


impact on the operations of the Service Bureaux pilot project. 


The resources and management areas of the Task Force, 
approaches to the training of staff, development of 
procedures/systems and general management of the Program 
Were .cxamined. The Jevel of mesourcing,..in global terms 
was recorded and the views of field staff were obtained on 
the adequacy and timeliness of Headquarters response to 


thein needs. 


The Task Force has mounted the Service Bureaux Program 
within a pressure-creating time frame. Further, the broad 
issues of training, equipping and directing the staff have 
been met, albeit by a combination of planned and ad hoc 
decisions. The issues requiring examination from a Program 
Evlauation perspective relate more to the continuing, 


future operation of the Program. 


Finally, the external dynamics of the pilot project vis a vis 
the Federal Departments and Agencies and Provincial/Municipal 
jurisdictions were considered. Under the terms of reference, 
however the study team was unable to explore these areas fully. 
Interviews conducted with Service Bureaux staff indicated a 
wide divergence of views and opinions. As the participation, 
attitudes and roles of these groups are crucial to the 
ultimate provision of service, the issues indentified and 


IncluoecrAn, this section are of .particular signifieance. 
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The issues identified in the Program/Related Dynamics 


auseay ait Cr: 


Issue #10 To what extent and to what effect have 


- Federal Departments and Agencies, 
- Provinctal/Muntctpal jurtsdicttons, and 
- Federal Members of Parltament and 


thetr,Constituency.0fftces 


played a role tn the Pitot Project inrtiatives? 


The Service Bureaux pilot project interacts directly or 
indirectly with virtually every federal department in 
providing access to information and expert referral services 
to the public. Interviews with Task Force members and 
regional visits confirmed the importance of this dynamic. 
Indeed, the training and operating guidelines of the 

Service Bureaux recognize the departments' position as an 


integral element in responding to public enquiries. 


Questions of fact requiring no interpretation or value 
judgement are responded to by the service personnel. More 
complex questions are carefully screened; the Client Service 
Officer probes to determine the real question and then 
telephones the appropriate government department/agency 

ang stays on’thes@inersuntil theright contactdis meechede 
The client is connected with the departmental expert and 


the enquiry is resolved. 


Interviews with field personnel suggested that while the 
response to the Service Bureaux was "positive" overall, some 
Client Service Officers perceived a "reluctance" and "a lack 
of commitment" on the part of some departments and information 


OLficers. 
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tThe-sererral role of the’ Service Bureaux is clear; it 
remains however, that the Bureaux operate in a sensitive 
environment. Evaluation of this issue is of importance 

and the findings would be of significant benefit in 
clarifying the linkages and relationships between the 
departments/agencies, Members of Partliament, Constituentcy 
Offices, and the Canada Service Bureau modules should 


the project become a continuing federal service initiative. 


Issue #11 How have tnternal management, tratning and 
communtcattons affected the operation of and 
acntevements of the Service Bureaux Program 


projects, both regtonally land. in total? 


Addressing this issue in the evaluation would provide the 
Task Force with an appraisal of the operating and organi- 
zational relationships and their impact on the achievements 
of the pilot project. For example, the majority of field 
personnel perceived a real need to communicate more 
frequently and more openly with Task Force management. 
Several suggestions were recorded including: 

~ regular meetings of the Regional Managers with 

Management; 


—- periodic meetings of Regional Managers/Satellite 
Bureau Managers from distinct geographic zones; 


- periodic regional seminars of client service 
OTrtrcers =" vand’, 


—- more frequent training and briefing sessions. 


In exploring this issue, evaluators would assess whether 
the internal management, training and communciations of 

the Task Force have beendelivered efficiently and to what 
extent the present arrangements have impacted on the 
effectiveness of the.Service Bureaux pilot.project.. The 
examination of this issue would also include an examination 
of possible alternative approaches where significant 


evidence indicated a need for change. 
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Issue #12 Is the level of and use of resourcing available and 


sufftctent to achteve the stated objectives of 


the program? 


Discussions with the Task Force management and regional 
personnel suggested that the level of resourcing available 
togthe project was,in itself a major area of concern. In 
addition, the reporting of and accountability for resource 


expenditures were also referenced. 


In addressing this issue, evaluators would identify and 
document the actual internal and external resources made 
available to the pilot project. The evaluators would then 
determine whether the order of magnitude of these initiatives 
was plausibly sufficient to ake the stated objectives of 
the Service Bureaux pilot project. By comparing the resources 
available to this experimental initiative with those of 

other projects/programs with similar intended impacts, a 
qualitative assessment could be made of the program Wiala— 
zation and focus of resources. Questions such as the 
possible effects of increasing e.g. scaling-up or decreasing 
Bureaux funding should be tested through sensitivity analysis 


techniques. 


Issue #13 Should a Program of provtding-aececessy referral to 
the publte be manned and operated through Publte 
Service staff or through conttnutng private 
contract arrangements and what are the dynamtes, 


benefits and drawbacks of each approach? 


The essence of the issue identified here is ‘related 

to the potential of the pilot experiment to become 

a continuing program initiative. With one exception, 

each of the Service Bureaux and Itinerant offices is 
staffed by private sector contract personnel. The use 

of contract personnel during this experimental phase 
permitted Task Force management to more quickly in mounting 


a credible enquiry and referral service. 
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Should the pilot project become a continuing federal 
service initiative, staff of the Service Bureau modules 
wiii*betdrawn from; the public ‘service .** The? questions 

to be considered under this evaluation issue centre 
first, on the dynamics of the continuing program and 
second; on the implications in mobilizing an effective 
public service staff of up to 300 personnel. Are 

there particular impacts on the service dynamics of 

the Bureaux, or particular costs/benefits evident at this 
time? Would management's activities in the administration 
of the Service Bureaux change e.g. staff relations, 


personnel administration? 


Bvaluecron Of This VWSSue LS ENeELehore” or Significant’ 

value in providing management with additional insight 

on the dynamics and viability of a continuing Service 
Bureaux program. Evaluation results, based on an examina- 
tion of the issue, would complement the pilot project's 
testing of the range of other Service related dynamics 

and respond to the questions highlighted by Task Force 
management and regional personnel in interviews with 


the study team. 


» 


Tesue $14 Do the office tocattons, tayouts, tmage portrayal 
and staff dynamtes tn handling the publie, have a 
bearing on the volume and types of public use 


and -to mnateefyeat? 


The dynamics of thas issue are am integral element of the 
Task Force Service Bureaux pilot program. The information 
obtained in the evaluation of -this: issue, would.addxress 
management's;need.for information on the adequacy of the 
design/enquiry demand determinants, the linkages to 
enquiry volumes and types of enquiries and could provide 
insight into alternative means of encouraging improved 


program/related client service dynamics. 
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ine tpnst AO fithe! padee. project is to test a number of 
service related dynamics. The variety of Service Bureaux 
locations and Itinerant Service areas are an integral part 
of this design. Regional visits by the study team confirmed 
the range of sites selected and the varying impact of each 


in terms of their design, projected image and service area. 


Evaluation of this issue would provide management with 
important insights in examining the impacts and dynamics 


issues of the pilot project experiment. 


5.5 Evaluation Issues Summary 


Below is a summary of evaluation issues. The program 


design issues are: 


Issue #l To what extent have Service Bureaux Program 
criteria been developed, adhered to and with 
what) current results for the future 


determination of operating modes and levels? 


Issue #2 To what extent has the Service Bureaux Program 
assisted in the determination of the market to 
be served and the actual levels of demand to 


be met? 


Issue #3 Has the development of data banks at local 
service levels contributed to the identifica- 
tion of the public needs for referral and/or 


access? 


Issue #4 Are the current interpretations and practices 
followed by the Service Bureaux Program 


adequate to assess the needs of Canadians? 
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The Service level issues are: 


Issue #5 is there evidence that the Service Bureaux 
Program has resulted in incremental changes 
in the volume of enquiries and changes in 
client satisfaction that would indicate a 
continuing need for, or modification of 


Program services? 


Issue #6 Are the dynamics of the Service Bureaux 


initiatives supportive of clients’ needs: 


——in Walk=an Oftices- 
~ an atinerant services: and 


—- in carrells? 


Issue #7 To what extent has the provision of publications 
been a factor in providing improved services to 


the: public? 


Issue #8 Is the current limitation of the Service Bureaux 
Program role (e.g. referrals only vs. referrals 
and information/in-person priority vs. in-person 
ands telephone enquiry .)! providingwva Suit icient 
level of service at the testing/experiment 


Stage? 


Issue #9 Has the level of advertising had an impact on 
the Service Bureaux Program operations and to 


what effect? 
The issues within the program dynamics area are: 


Issue #10 To what extent and to what effect have 
- Federal Departments and Agencies, 
- Provincial/Municipal jurisdictions, and 
- Federal Members of Parliament and their 


Constituentcy Offices 


played a role in the Pilot Project initiatives? 
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Issue #11 How have internal management, training and 
communications affected the operation of, and 
the achievements of, the Service Bureaux 


Program projects, both regionally and in total? 


te5uc 7i2z is the level of and ‘use of resourcing available 
and sufficient to achieve the stated objectives 


of the Program? 


issue #13 Should a program of providing access/referral 
to the public be manned and operated through 
Public Service staff or through continuing 
private contract arrangements and what are the 


dynamics, benefits, and drawbacks of each approach? 


Issue #14 Do the office locations, layouts, image 
portrayal and staff dynamics in handling the 
public, have a bearing on the volume and types 


of public use and to what effect? 


The wide range of issues covered, as indicated earlier, 
have significance for the types of evaluation that could 


be considered by Task Force management. 
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6.0 EVALUATION OPTIONS 


Gl Rationale for Evaluation Options 


The study team has developed five (5) evaluation options 
which address the range of issues identified during the 
Evaluation Assessment. An effort has been made to select 
options that, in examining the dynamics and processes of the 
pilot project, will provide management with increasing in- 
Sight into the critical components of the Service Bureaux 
experience.. A significant factor in developing the 
framework for these various options was the formative stage 


of the Service Bureaux pilot project at,this time. 


The five (5) basic evaluation, options presented in this 


Section are: 


An Ex-Post Evaluation Option: which would measure 
efficiency and effectiveness to date and relate the findings 
to the potential of the Service Bureau Program to achieve 
stated objectives, including assessments of the effects of 


increasing or decreasing program resource levels. 


An Effectiveness Monitoring Option: wherein a 
determination would be made of the extent to which established 


objectives and desired impacts were achieved. 


A Management Monitoring Option: in which the design, 


process, criteria and data structures and the management 
directives and actions would be evaluated to determine the 


degree to which they contributed to program achievement. 


A Formative Evaluation Option: involving intensive 


evaluation of all aspects of the Program to determine the 
revisions needed to ensure the relevancy of objectives and 
processes, the efficient use of resources and designs for 


alternative means of program delivery. 
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An Evaluation for an Experimental Project: in which the 
focus of the evaluation would be the more immediate operational 
needs of management in resource planning, internal processes/ 
data management and administrative improvements and the important 
area of assessment criteria for the ultimate analysis of and 
decisions on the design and operating modes for a continuing 
program of referral and access to Federal Government services. 
The rationale for the establishment of the experimental project 


would not be tested. 


A cost breakdown for each option is provided in each of the 
following descriptions of the options. The total costs identi- 


fied include consulting fees, survey and other expenses. 


Gs2 An Ex-Post Evaluation _Option 


6.2.1 Scope 


The scope of the Ex-Post Evaluation Option is to examine the 
relevancy of the program's objectives, causal links and 
processes and to measure the efficiency and effectiveness of 


the program's processes and actions. 


Gl2ee PFUurpose 


The purpose is to determine; 


= the plausibility of “the Zinkage of objectaves to 
activities and effects; 


—- the intended and unintended impacts of the program; 
—~ the levels of efficiency and effectiveness achieved; 


= the degree of duplication, overlap or conflict with 
other programs; 


—~ the effect of discontinuing, increasing or decreasing 
the resource levels; and 


- the possible strengths or weaknesses within the Program. 


The findings of the Evaluation would provide management of the 
Task Force with an understanding of the critical Program linkages 
and the performance of key components of the Pilot Program. 

This information would contribute to the determination of 


appropriate future thrusts and levels of activity for the Program. 
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6.2.3 Methodology 
The methodology, in capsule form would involve: 


—- an internal detailing of data resources, processes, 
criteria, performance indicators, resourcing and 
an assessment of cach OL the above’ elements; 


— an extensive survey of client areas to determine 
client satisfaction levels; 


—- an intensive series of interviews with all interacting 
programs and services; and 


—- detailed cost-effective analysis of alternative 
levels of resourcing. 


6.2.4 Estimated Duration and Study Level 


The study would involve 4 - 5 months of detailed,,concentrated 
work, involving a minimum of three consultants, and. several 
survey/questionnaire approaches. The estimated cost would 


be in the 50/60 thousand dollar range. 
6.3. An Effectiveness Monitoring Option 


6.35.4 Scope 


The scope of Effectiveness monitoring Evaluation would be 
to determine to what extent were the established objectives 


of the Program and the desired impacts achieved. 


bee, | PuBpoSse 


The purpose of Effectiveness MONTtOring 12s co tes tie: 


—~ effectiveness measurement data from Program and users; 
- objectives Criteria: 
—- cost indicators; and 


- other possible impact results. 


The findings would be used to present the current achievement 
levels of the program and their applicability for the future 


activities of the program. 
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6.3.3 Methodology 


The methodology would involve: 


- detailing the program measures of effectiveness 
and their relation to the objective and desired 
impacts; 


- testing through review of existing data and survey 
users, the validity of performance data; 


- assessing the relative cost effectiveness of the 
results obtained by cost/benefit analysis; and 


- conducting analysis of the proven results against 
program criteria. 


6.3.4 Estimated Duration and Study Level 


The monitoring of effectiveness would involve 2 to 5 months 
of two consultants and limited survey work. The estimated 


cost would be in the 25/30 thousand dollar range. 
6.4 A Management Monitoring Option 


6.4.1 Scope 


The scope of this Evaluation Option is narrower and primarily 
targeted at the efficiency areas of the program, but with an 
examination of the management policies and practices as they 


relate to program effectiveness included. 


O62) Purpose 


The purpose is to determine within the accepted objectives, 
causal links:and: existing processes: 
- the degree to which resources and program processes 
were utilized within planned levels of activity; and 


- the extent to which management control and direction 
affected the achievement of results. 


Such an option would provide valuable information to manage- 
ment and evidence/direction for improving the pilot program's 
effectiveness. The evaluation approach adopted would be 
Significantly more quantitative than the methodologies 


described in the previous options. 
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6.4.3 Methodology 


The data requirements of this evaluation option are more 
extensive and the analysis required more rigorous than 
was possible during the Evaluation Assessment. The methodo- 


logy would encompass: 


- systems design analysis; 

—- measurement data usage analysis; 

—- control mechanism assessment; 

~ anterviews of staff at all levels; 


—- detailed analysis of budgets, costs and utilization; 
and 


—- an analytical comparative review of Program plans 
and performance results. 


6.4.4. Estimated Duration. and Study Level 


The study of Program Management would involve 2 to 3 months 
of work, involving two consultants, travelling to regional 
service bureaux offices for interviews and file/records 
search and analysis. The estimated cost would be in the 


25/30 thousand dollar range. 


6.5 A Formative Evaluation Option 


O65, bay aecape 


Formative evaluation is an intensive evaluation option 
iMvolving all “aspects to® the Pilot Program.” Alternative 

models for program delivery are stressed including the reframing 
of objectives, criteria, processes, data’ base and activities as 
deemed necessary to ensure the continuing relevancy of 


objectives and processes. 


Boo. e  PUrpose 


The purpose of formative evaluation is to determine the 
revisions needed to the program's objectives, causal 
linkages and processes to ensure their relevancy, the 
efficient use of resources and to ensure the impacts and 


effects meet desired levels. 
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Os Pe Methodology 
The methodology, in brief, would involve: 


—- extensive research of Task Force files, documents 
and decisions to analyse the retionale and the 
linkages of the program; 


- an intensive detailing of processes, criteria, 
data sources, performance indicators, resourcing 
and assessment of these elements; 


- an intensive survey of users, interacting agencies 
and departments to determine effectiveness of the 
program; 


—- examination and analysis of other similar programs 
in Canada and elsewhere to determine potential 
alternative program delivery designs; and 


—~ detailed modelling and cost implication analysis 
to determine the effects of various, design/delivery 
proposals. 


Lr 


6.5.4 Estimated Duration and Study Level 


The study, by its intensive nature would take 5/6 months 
and would involve both consultants and technical design 

experts and extensive survey/interview time and coverage. 
The estimated costs would be in the 60/80 thousand dollar 


range. 


6.6 An Evaluation for Experimental Projects 


6.6.21 Scope 


The evaluation framework proposed for this experimental 
project would emphasize criteria used to establish the 
various alternative approaches and the means of capturing 
and analyzing’ Gata’ from each approach to ensure’ that Task 
Force Management is equipped to advise the Government on 
the optimum approaches for providing access to Federal 
Services. The immediate management and process issues in 
administering these operations would also be evaluated. 
Elements of management's practices and meeting of objectives 
would be reflected in examining the challenge of and the 
response to bureaucratic processes - a function normally 
reported through management audit studies - to determine 
the effects on overall Service Bureaux achievement of 


objectives. 
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6.6.2 Purpose 


The purpose of this evaluation framework approach is to 
provide management with critical feedback on the key 
processes/activities and linkages within the experimental 
project design. The thorough and consistent basis for 
defining and refining the pilot study over time provides 
direction to the present and future management of the 


Service Bureaux experiment. 


6.6.3 Methodology 


The evaluation methodology of this experimental project 


would encompass: 


—- extensive research of Task Force files, documents 
and decisions to examine the immediate operational 
issues e.g. the suitability of bureaux design and 
location; the effect of signage and projected 
Service Bureaux image. 


- an intensive detailing of processes, criteria, 
data sources, performance indicators, resourcing 
and assessment of these elements; 


—- interviews with staff at all levels e.g. staff 
approaches to the various publics; contract versus 
public service staff, immediate organizational 
issues. 


—- analysis of budgets, costs, utilization and the 
availability of comparative data e.g. the objectives 
achievement and impacts and effects evaluation issues. 


-—- Survey ‘of users, interacting agencies and departments 
e.g. efficiency and effectiveness of the experimental 
design. 


—~ detailed comparative analysis of each of the variety 
of approaches adopted by the Service Bureaux project. 


—- detailed modelling and cost implication analysis 
to determine the effects of various design/delivery 
limitations for a future continuing program. 


- testing the current capacity for analysis of the 
variety of approaches and assisting in the design 
of the analysis framework if required. 


—~ examination and analysis of other similar programs 
in Canada and elsewhere to test the approach and 
design elements used by the Task Force. 
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6.6.4 Estimated Duration and Study Level 


The focus of the evaluation on the operation issues of the 
experimental program design would be 3/4 months duration 
including survey and intensive time. The estimated costs 


would be in the 30/35 thousand dollar range. 
637 Criteria for Assessing Options 


In order to objectively compare the evaluation options and 
assist management in selecting a preferred alternative criteria 


are proposed consisting of: 


is The need to know. Cabinet and Task Force management 
require a 'prototype'’ evaluation to provide interim answers 
Lo ensure that the existing Service Bureaux Pilot Project 
design is "on-track' and will provide appropriate answers to 


the service and dynamics issues of enquiry agencies. 


2. Highlighting of the key expected results which would 


emerge from the evaluation. 


=e Required historical data base in the conduct of the 
evaluation. 
AY The expected calendar time required to complete the 
evaluation. 
5. The expected level of effort required to conduct the 
evaluation. 


Exhibit .9 provides a summary of evaluation options by the 
assessment criteria permitting management decision on the 


most appropriate option for their needs. 


The level of effort required to conduct each evaluation 
option was estimated assuming a study team comprised entirely 
of consultants. Should Task Force/Departmental staff 
participate, the indicated levels of effort may be subject to 


revision. 
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6.8 Recommended Evaluation Option ~ 


The Ex-Post and Formative methodologies offer intensive 
evaluation options. While perfectly valid even at this early 
stage in the experimental project's existence, there are 
important disadvantages associated with these approaches 
that outweigh the obvious advantages of seriously examining 
the pilot project and re-orienting it as early as possible. 
The major and overriding disadvantage is the lack of 
sufficient performance data and historical context to 

either support or refute systems change at this time. If 
however, management were to consider deferral of the 
evaluation for any period exceeding one year, then either of 
the options should be considered. Further, the selection 

of an evaluation option to meet urgent reporting time 
constraints now would not preclude the conduct of either 


of these evaluation options at some later date. 


The Effectiveness and Management Monitoring options have 

the distinct advantages of providing management with essential 
feedback from aspects of the pilot project experience on 
which the decisions concerning future, funding and activity 
leveis may be based. The disadvantages of these approaches 


are their reliance on the availability and reliability of 


data as well as, the degree of difficulty presented due to 
the relatively brief life span of the Service Bureaux pilot 


project. 


The study team recommends that the option entitled Evaluation 
of an Experimental Project be used in an evaluation of the 
pilot project experience at this time. The advantage of this 
evaluation framework is the option's greater sensitivity/ 
responsiveness to the more immediate needs of management in 
the areas of resource planning, internal operations processes 
and managerial improvements. As well, the option seems 


better suited to an experimental project design that: 


- has recently been introduced; 
- has a varied range of approaches; 


- is designed to provide answers, but also provide 
service; and 


- is flexible and adaptable to changing pilot project 
design. 
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Despite this recommendation and the study team's 


presentation of evaluation options, the Task Force has 


the ultimate responsibility for selecting and/or 


redefining the proposed evaluation options. 


629 Evaluation Study Terms of Reference 


(Based on 


the assumption that the Task Force Management 


selected the Experimental Project Evaluation) 


The Experimental Evaluation Study Team should: 


focus on the design of the Program in terms 
Of Ats ability to provide 4 represenative 


variety of service alternatives; 


assess the criteria used to ensure that design 
elements were adopted for each mode and variety 


of services; 


determine the particulars of the data base, its 


sources and the uses to whach it is utilized; 


examine all supporting dynamics to the Program, 
such accommodation, location and design, 
training, equipment, stati relations, etc., to 
determine their degree of supportiveness to 


the Program; 


determine, through sampling, the level of 
effectiveness of the experimental Services, as 
an aid to assisting Task Force management 


in determining ultimate service levels; 


determine, through sampling, the source of, and 
reason for, the clientele, as an aid to market 


penetration determination; and, 


gauge the degree of support or concern existing 
within Departments, MP consistency offices and 
other affected areas, on the efforts of the Task 


Force. 
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APPENDIX A 


SERVICE MODULES 


AND 


DESCRIPTION 


BACKGROUND 


This appendix presents a brief overview of the components 


of the Service Bureaux pilot project. 


DSS Task Force on Service to the Public (1978) 


The current Task Force on Service to the Public was 

formed to work with all departments in raising the quality 
of government service, to improve communications between 
the citizen and the public service and to make it easier 
for people to deal with the government. The three target 
areas identified by the Task Force for action are therefore 


Service Delivery, Awareness and Access. 


The Task Force's Service Delivery and Awareness work 

is in its formative stages but, the object is to ensure 
that public servants do a more effective, meaningful and 
satisfying job. The employee responsible to the public 
will no longer be limited to the person across the desk 
or working on the switchboard. Task Force activities are 
directed to ensuring that all public servants will be 
involved in and accountable for serving the public and 
are equipped to help, to explain or listen to the public 


by phone, in person, or, by. letter. 


Access to Government Services~and. Information 


Since its creation, the Task Force has focused considerable 
attention on attacking the problems that Canadians have in 
accessing and obtaining Federal services and information. 


Tie acrack was. launched on threes Eronts = -—= 


. Access by telephone 
..access through in person visits; and, 


. Access through Members of Parliament. 
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The Service Bureaux pilot project is directed to improving 
the public's access to federal programs and services through 
the experimental examination of in-person enquiry facilities. 


The objectives of this approach are: 


- To determine the level of service, the cost 
‘effectiveness and market penetration levels 
required to meet the needs of Canadians; and 


- To find ways and means of attacking the problem 
and to develop an approach to solving it. 


(Cabinet Document, 
July 1980) 


Access Through In-Person Visits 


The Treasury Board Task. Force study in 1977 into citizens’ 
perceptions of government indicated that there was an apparent 
widespread dissatisfaction with the quality of federal services. 
The study also suggested that many respondents preferred to 
deal with government on a face-to-face basis. Even in 
government, personal contact is sometimes better than the 
telephone when discussing the completion of forms, looking 

up pamphlets or discussing correspondence which may be 

causing problems. The majority of those who preferred personal 
contact in the TB Study group were younger Canadians and 


those who lived in smaller cities and rural areas. 


The federal presence in small communities is virtually non- 
existent. In small cities it exists however, but is 
fragmented in such a way that a citizen can easily become 
extremely frustrated inifinding’ the "right”> place? ton deal 
with an enquiry. The problem is further compounded as 
Canadians are ill-informed about jurisdictional responsi- 
bilities, either by department or by level of government. 
Even in major centres, where there are federal buildings and 
activities, the problem of serving the needs of the public 
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The Service Bureaux Pilot Project 


Studies and surveys have continued to demonstrate that 
the public is concerned in its dealings with the large 
and complex federal organization. Despite the enormous 
number of enquiries received by the government every day, 
no co-ordinated or consistent system existed to handle 
this traffic. The introduction of the Service Bureaux 
program is an attempt at improving the Federal 
Government’'S enquiry contacts with the public. 

The Service “Bureaux Pilot “Proyect is’ ‘comprised of four 


modules: 
Service Bureaux/Satellite Bureaux 


Bureaux are street level facilities located in a variety of 
high-traffic locations (e.g. existing government buildings, 
commercial/financial districts, heritage buildings, shopping 
centres and along major thoroughfares); they are situated 
and designed so as to encourage the public to enter, browse, 
Sit and read; equipment and resource materials include the 
telephone, reference books, government publications, 
directories, indexes, pamphlets...; the public uses the 
services provided by the Bureaux to gain basic information 


° 
or obtain a referral to a government agency. 


Ttinerant Officers 


Ttinerant officers are part of a mobile enquiry service that 
covers smaller urban communities and portions of the rural 
areas of Canada. Working out of a base office and reporting 
to a walk-in service bureaux, the Itinerant Officers travel 
a regular route on a pre-arranged timetable carrying 
necessary reference material and using local telephones as 
backup. The services provided by the Itinerant Officers 

are basically identical to_ those provided by the Service 


Bureaux/Satellite Bureaux.” 


1. The Task Force is currently examining the potential of 
equipping the Itinerant officers with portable Telidon 
capabilities. 
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For the purpose of the pilot program, 8 mobile offices 
are based in Kelowna, Gimli, North Bay, Sherbrooke, 


Matane, Bathurst, Sydney and Windsor (Newfoundland). 


Information Carrells 


Carrells are unmanned free-standing units designed to present 
reference materials on federal programs and services and 

in 30 proposed locations, a Telidon video display screen. 

The carrells will be placed in prominent sites (e.g. shopping 
malls, Post Offices, Cultural Centres, Employment Centres...) 
where the public will be able to use these units by inter- 
acting with the Telidon system or by examining the reference 


literature displayed. 


The carrells are maintained and serviced independently by — 
the nearest Service Bureaux or under separate agreements 


with sponsoring groups. 


An additional sixty (60) non-Telidon information carrells 


are planned for implementation in the 1982/83 fiscal year. 


APPENDIX B 


STUDY WORK PLAN 
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Revised August 19 
WORK PLAN SCHEDULE 


1. Confirmation of Work Plan, - 
Management Review August 19 
2. Conduct Key Interviews August 20 - 24 
3. Develop Program Profile August 24 - 28 
4. Develop Context Model August: 24 0—' 26 
5. Develop Assessment Strategy ANUSUSE 20) =, Sept 
GG. Present 3, 4 and) 5 FOr 
Management Review September 2 
7. Conduct Assessment Interviews September 2 — LL 
8. Assess Program Structures 
and Context Model September 8 —- 11 
9. identify Evaluation Issues september 8 — Ll 
10. Present Revised Models and 
Evaluation Issues for 
Management Review September 14 
11. Develop Evaluation Options September 14 - 16 
12. Develop Evaluation Options' 
Criteria and Costing September 14 - 16 
13. Draft Assessment Report — 
Management Review (Selection 
Of Option) September 17 


14. Develop terms of reference for 
full evlauation and draft final 


report September 18 - 
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Mon) Eresencation of Final Report September 21 


APPENDIX C 


MINUTES OF STEERING 


COMMITTEE MEETING 
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1 Ap as ie 
Minutes of First Meeting MP eaVguee S . 
Services to Public Steering Committee 
In attendance: Sackmann - Task Force 
McDougall - Task Force 
Robert -— Task Force 


Maffini - DPA 
Smith - DPA 

Nelson —- DPA 
Burkart - DPA 


DAUD QAPVP 


The agenda was tabled and the Work Plan draft reviewed. 


ax. Was agreed that: 


- DPA would review the Work Plan 
schedule to determine 1£ the time 
frames could be tightened by at 
least one week. (NOTE — this was® 
done and a. Work Plan covering 
August 20th to September 21, 
provided to T.F. an August 21.) 


Mr. Maffini stressed that the Evaluation Assessment would 
identify what could be evaluated. Robert asked if 
both a short term view and a longer term view could be 
. 
taken with an evaluation framework and criteria developed - 


AOTeecda. 


Mr. Smith raised assuie of the need: for interview lists = 
both first round key interviews and second round names. 
Also Mr. Maffini asked abovt a DM Interview. Le was 


agreed that: 


— Mra McDougall would meet eDPA — 20 
August ~— 10 A.M. te develop ists 


—~ Mr. Sackmann would check on DM interview 
(NOTE - Lists prepared with McDougall - D 


not available for interviews). 
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Mr. Maffini raised OCG representative participation - 
Mr. Sackmann agreed to check. (Note - OCG Representative 


will Participate | t@ ibe. named) . 


Mr. Robert proposed that Task 12, Development of Options 
should be done in conjunction with Task Force members - 


Agreed. 


On the question of attempting to assess and ultimately evaluate 
such new (and in some cases, not yet implemented) initiatives, 
it was agreed that, notwithstanding the problems, the 

study team would attempt to assess potential and concentrate 


on future evaluation criteria and framework, 


Mr. Sackmann raised several points relevant to the focus 


of the assessment study: 


—- the physical design location and 


impact of Bureaux needs to be assessed; 


- the emphasis on Telidon in DPA 
proposal should be downplayed in 
the study; 


Y =« the awareness of the various. publics 
of the Bureaux initiatives shovld be 


examined. 
Tt was agreed that these points would be noted and Mr. 
Maffini stated that an on-site poll technique would be 


used at Bureaux locations to test public reaction. 


The next Steering Committee Meeting is scheduled for September 2. 


U Ate’ tenes oF paesoeee, Pe Ye OH 
" : AS 


Ortigas... Pay Papin | eeematigns 2 amt tee ‘Bice, Wei 
; , eae ~ vel 


Cie ; es 
ices Loa ARE ae 
ee as 


x 
} 
73a wyren Force ae 
~~ 1 4a oe % 45 r) 
f L 
_ i 
’ ; 


AR Ua v3 ar i" ime i x! set 
Seanp. 
A i 


Sehr: ah 3 ne Dh fae er Dd, ALI meee 


Pi f 45 2 ¥" 
4 a, 
i sie I 
; $ y 
i ; 
: 
. a ' 
ie 
. ’ ‘ 
{ - 
, é > 
; Sf j ? 
* . , 
¥ iMG 
i 
j - ee 
+ , 
oi 
‘ : ao) 
' « b 
. rt a @ 


| Siw Otnting sath -dsd4 


‘ioe? Ite ot eese is, Vass 
i a al 7 = mo « y 


et wits A 


22 September 1981 


MINUTES OF THIRD MEETING 
SERVICE TO THE PUBLIC STEERING COMMITTEE 


in attendance: “R. August —"Task Force 
A. Sackmann - Task Force 
P. Macdougall - Task Force 
P. Trudel - Task Force 
J. Batchelor -— Comptroller General 
G. Maffini - DPA 
Bb. Smith — DPA 
D. Nelson — DPA 


The purpose of the meeting was to review the Evaluation 
Issues and Options identified. It was agreed that the 
Evaluation Issues expressed in the Interim report covered 
the range of concerns of significance in evaluating 


the Service Bureaux Pilot Program. 


Four evaluation approaches were presented to the Task 
Force for consideration. Discussion focussed on the 
vossible distinctions between program and project evalua- 
tion. it was decided that a fifth option embracing the 
unique experimental role of the pilot project would be 
developed (Evaluation of an Experimental Project). 
Further, it was decided that the methodology of this 
fifth option would emphasize the probing of issues to 
ensuxe that the experimental program was on track. and 
secondly, that the benefits of the program are evident 


and related to overall goal/target achievement. 


It was agreed that evaluation of the Service Bureaux pilot 
program at this time would not preclude a more extensive 

evaluation of the Service Bureaux concept at a later date. 
Indeed, it was agreed that the staged evaluation approach 


would be very beneficial. 
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The next Steering Committee meeting is scheduled for 
29. September to review the draft final report on the 
Evaluation Assessment of the Service Bureaux pilot 


program. 


APPENDIX D 


INTERVIEW CONTACTS AND 
GENERAL FORMAT OF 
INTERVIEW GUIDE QUESTIONS 
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INTERVIEW CONTACTS DURING THE EVALUATION ASSESSMENT 


The following interviews were conducted with Task Force 
and Departmental personnel during the Evaluation Assessment 


of the Service Bureaux pilot project: 


Task Force Departmental 
R. August R. Davis 
A. Sackmann (DSS, Personnel) 
P. MacDougall 
P. Tradel 
REGIONS 
Winnipeg Quebec City Interlake, Manitoba 
G. Hall J. Babin F. Rankin 
C. Duges L. Garant W. Dalman 
S. Forbes H. Maheux 
C. Scerbo 
Edmonton Montreal Sherbrooke, Quebec 
D. Savaria L. Boisvert G. Roy 
B. Cameron L. LaRiviere 


N. de Lorme 


J. Lacoste 
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INTERVIEW GUIDE QUESTIONS 
A. Roles/Responsibilities 


1. Name, position, title? 
2. Describe role, duties, responsibilities? 
3. What contacts do you have with the Task Force? Reporting/ 
. Accountability links to the TF? Who is the most frequent 
contact? 
4. How do you perceive your role vis a vis the in-person 
devices offered in the pilot program? 


5. Duties - % distribution of time? 
E.G. °S 
Mgr CSO Clerical 
Mgmt 30 Maintenance 20 Admin. 60 
Liaison 30 Referrals 60 Stats. 30 
Referrals 20 Expt 20 Relief 10 


Liaison other 20 


B. Current Status of Program 


1. What is your understanding of the overall plan of action 
for the pilot program? 

2. What is your understanding of the intent of each of the 
components? 

3. Interface/relationship? 
a) DSS 
b) Dept's - reaction to date 
c) Media/Local service agencies 
d) other ievels of government (provincial/municipal) 
e) MP's and constituentcy offices 
£) Consulting firm 

4. Is information relating to budgets, cost of implementation... 
available? How is the budget handled? Invoices? 

5. Are you aware of performance measures to monitor volumes, 
satisfaction levels, effeciency? Describe. 

6. Effects of relays in implementation, delivery of 
equipment, staffing...? 


C. Expectations/Products 


1. Understanding of a) objectives, and 
b) products 
of the program? 
2. Awareness of effects or impacts either intended or unintended 


to date? 
3. Relationship of in-person visit pilot project to other TF 


initiatives (e.g. telephone access, blue pages...) 


D. Evaluation Issues 


1. Views on key evaluation issues? Rationale? 

2. Views on means of determining relevancy & evaluation of issues? 

3. Persons or organizations that are directly related to the issue 
identified? 
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E. Service Bureaux/Itinerants 


I Operacion 6 dave, Official opening) & date? 

2. Number. of personnel, full time/part time? 

3. General nature of requests for information? 

4. Are records maintained of volumes? Are monthly enquiry 
figures available? 

5. Typical enquiry response process? Follow-up? 

Go Prorile of a typical client? 

7. Profile of a repeat user? 

8. Perceived level of client satisfaction? 

9. How is data bank structured? Adequacy of data banks 
e.g. breadth of santo structure? 

10. How would you rank public awareness of your service/ 
of the program? 

id. ideal personnel characteristics of SB statf£? 

12. Adequacy.of Service Bureaux as to design and location? 
era. "porential | Lor ciseplays, pack-uplatter Nourse? “Are 
you familiar with the criteria on Servace Bureaux placement? 

13. Possible improvements to the system? 


is the rotation of staff from the Service Bureaux to the 
Ttinerant Service useful? Explain? 


All Anterviews 


- provide a summary of issues and effectiveness to date. 
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APPENDIX E 


LIST OF SOURCE DOCUMENTS 


REVIEWED 


LIST OF SOURCE DOCUMENTS REVIEWED 


Reports, Studies and Correspondence 


The following source documents were reviewed in the 


course of the Evaluation Assessment: 


ae 


ae 


CoS 


Lo. 


Cabinet Discussion Paper Task Force on Service to the 
Pubpiac. July, 1930. 


Cabinet Discussion Paper Service to the Public 
Panuaryi 24 Lode . 


Miscellaneous Cabinet Document Working Papers. 


Research Report: Task Force on Service to the Public 


(Treasury Board) March, 1978. 


General correspondence and briefing notes to the 
Minister, Supply and Services Canada re: Task Force 
Status (19380, 1981). 


Minister's Main Estimates Briefing Book, 1980-81. 


Record of decision. Cabinet Committee on Communications 


(25 February, 1981) . 


Speeches by the Minister on opening of miscellaneous 


Service Bureaux. (1981) . 


Miscellaneous Service Bureaux promotional and official 


opening kits. 


Task Force on Service to the Public reference manual. 
Principles of Service, The Basic Service Model, The 


Service Improvement Model (February 1981). 


Reference Manual for the Client Service Officer 


(April 1981). 


Miscellaneous contract data for consultants, supplies 
and services in establishing the Service Bureaux pilot 


program. 
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13-- Mandate and minutes from meetings of the Advisory 


Board to tne Task Force-on Service to the Public. 


24. Parliamentary Returns on Service to the Public issues, 


ace. , the Location of Federal Service Bureaux. 


15. Miscellaneous Task Force correspondence re: exhibitions/ 


displays, Members of Parliament. 


ios mepor (by di. Peli.) Service, tq the Public, A Pilot 


Program for Mobile Service (January 31, 1981). 


17. Report on Canada Service Bureau. “Winnipeg, Manitoba. 
UMLY 292". 


18. Miscellaneous status and activity reports from the 


Canada Service Bureaux. 


19. Rapport sur les bureaux itinerants. Gilles Roy 


le 8 septembre, 1981. 


90. Guide on the Program Evaluation Function Office of the 
Comptroller General of Canada, May 1981. 
9)... Program Evaluation: An Introduction. Office of the 


Comptroller General Of Canada, Draft December 1980. 


22. A guide on the Program Evaluation Function in Federal 
Departments and Agencies. Office of the Comptroller 


General of Canada, Draft November 1980. 


23. Principles for the Evaluation of Programs by Federal 
Departments and Agencies. Office of the Comptroller 


General of Canada, Draft December 1980. 


Note: Detailed financial data on the status of the Service 
Bureaux pilot program was not available to the Study 


Team. 
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APPENDIX F 


INTERIM REPORT ON 
EVALUATION ISSUES AND OPTIONS 


SERVICE BUREAUX PROGRAM 
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INTERIM REPORT ON 
EVALUATION ISSUES AND OPTIONS 
SERVICE BUREAUX PROGRAM 
TASK FORCE ON SERVICE TO THE PUBLIC 


18 September 1981. 


DPA Consulting Ltd. 
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Ls INTRODUCTION 


har ot 8 Purpose 
The purpose of this report is twofold: 


a) To present the evaluation issues and evaluation 
options resulting from the findings of the 


evaluation assessment study; and 


b) To present, for discussion and decision, dan 
alternative approach to determining the effective- 
ness of the Service Bureaux Program, based on a 
Project Evaluation approach rather than Program 


Evaluation. 
\ ages The Problem 


Program Evaluation, as described within the guidelines of 
the Office of the Comptroller General, is aimed primarily 
at evaluating on-going programs where an historical back- 
ground of data is available and where the rationale for 
the program and its approach needs to be evaluated. 

The Service Bureaux Program was established, as a pilot 
project only, to assist the Task Force to meet the Cabinet 
requirement "to determine the level of service, the cost 
effectiveness and market penetration levels required to 


meet the needs of Pankdvana 


In this context, as a pilot project, the Service Bureaux 
Program was developed to find ways and means of "attacking 

the problem and to develop an approach for solving $e" 

and not as a long-term continuing Program whose sole objective 


would be provision of services. 


1. Discussion Paper, "Task Force on Service to the Public” 


dated July, 1980. 
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The problem from an evaluation perspective is which 


evaluation approach to take. 


If the Service Bureaux had been in operation over an 
extended period and a level of data and assessment 
findings had been developed, the major evaluation issue 


would be: 


"To what extent has the Service Bureaux 
Program assisted in the determination of 
levels of service, cost effectiveness and 
market penetration required to meet the 


needs of Canadians?" 


At this early, formative stage of the Service Bureaux 
rogram development, however, a very different major 


evaluation question, needs to be asked: 


"Are the current program initiatives, 
processes and data frameworks suitably 
designed and executed to ensure that the 
efforts of the pilot projects will provide 
a sufficient range and depth of information 
to enable the determination of levels of 
service, cost effectiveness and market 
penetration required to meet the needs of 


Canadians?” 


ce Decision Required 


It is the view of the Study Team that: the differences 
between these two basic questions are the differences 


between a Program and Project approach to evaluation. 


A decision of the Task Force management on this issue is 
required for the final framing of the Assessment Evaluation 


Report. The draft final report, containing a more detailed 
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account of the study team findings has been developed. 
Once the management decision on the approach to Evaluation 
has been determined, the final report can be submitted 
with the terms of reference for evaluation and other 


supporting detail included. 


The following sections include: 


Section. 2.—- Evaluation Issues 
section, 3,.—- Program Evaluation Options, and 
Section 4 = A Project. Evaluation Alternative 


Approach. 
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2%) EVALUATION ISSUES 
css We Introduction 


Within the current guidelines and approach to evaluation 
issues, as supported through the Office of the Comptroller 


General, four main issue areas are highlighted: 


- Program Rationale; 
- Objective Achievements; 
- impacts and, Eitects:: and 


— Alternative Program Delivery. 


The evaluation issues presented, cover all four evaluation 


areas Of Concern. 


In discussion with Task Force management, however, other 
areas of concern were voiced, which have been included. 
The line between evaluation and audit-type issues may 
appear, therefore, to have been crossed in a number of the 
issues identified. The decision to include these issues 
was predicated partly on an awareness of the need to meet 
a Cabinet direction and partly in the light of our later 
recommendations for project-type evaluation. 


e 


Lee Issues 


2.2.1 ‘Design Issues 


The main objective of the Service Bureaux Program is to 
determine the level of service, the cost effectiveness and 
the market penetration levels required to meet the needs 


o£ Canadians. 


Based on this objective, the Task Force designed a program 
to enable the operation of and assessment of service 


approaches. It was anticipated, for purposes of the 
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Evaluation Assessment Study, that the Service Bureaux 
offices and other services under the program would be 
designed, located and operated, not so much to provide 

a pre-determined level of service, but to provide a 

range of different services. The purpose of this would 
be to determine the proper mix of locations, levels and 
_types of services for future decisions to be made based 

on their findings. In order to achieve this objective, 
the design of the Program should be predicated upon clear 
criteria and carefully constructed recording and reporting 


mechanisms for future assessment. 


The study team did find evidence of a variety of locations, 


of service approaches and information recording methods. 


Ortice iocations were varied but a significantuiquestion 
emerged as to whether Program criteria were modified or 
ignored in order to obtain space quickly and become operative. 
The role of Public works and the level of their response 
to Program needs and the impact of this on Program Objectives 


should be examined. 


In the area of services provided, the study team found a 
variety of service approaches. The value of each different 
type of approach should be the provision of comparative 
results. Whether the means to assess service approaches 
exists and to what extent it is meaningful would form a 


key part of a subsequent evaluation. 


In a Similar vein, all offices visited maintained records 

on numbers and types of visits and enquiries. No clear 
evidence was found on whether the information forwarded to 

the Task Force was used, or that formal assessment formats 
were used to provide findings. The basic premise of operating 
the Service Bureaux Program is to utilize such findings and 
therefore the inclusion of this aspect of the Program in an 


evaluation rs evident. 
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The issues to be resolved within the program design areas 


abe. 


Issue #1 To what extent have Service Bureaux Program 
criteria been developed, adhered to and with 
wnat’ current results “and for the future 


determination of operating modes and levels? 


Issue #2 .To what extent has the Service Bureaux Program 
assisted an. the determination of the market to 
be. served and the actual levels of demand to be 


met? 


Issue #3 Has the development of data banks at local 
service levels contributed to the identification 


of the’ public needs for referral and/or access? 


Issue #4 Are the current interpretations and practices 
followed by the Service Bureaux Program adequate 


to assess the needs of Canadians? 
2.2.2 ‘Service Level Issues 


While it is ‘recognized ‘that ‘the major objective as to 
identify ways and means of ultimately shaping a Program to 
meet Canadian needs, “The effort to produce'tas viable a 
level of service as possible is also a significant part of 
tie Program effort =" Only ‘through Palie’ best! sex Vice SirorEs 
at the Pilot Project stage, can improvements or need to 


change be assessed honestly. 


Tt is therefore within the scope of the evaluation to 


examine issues of current effectiveness of operations. 


The impact of the- Pilot Project modes on the public was 
impossible to examine within the limits placed on personal 


interviews at the Evaluation Assessment stage. Obviously 
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unless some incremental change in the level of public 


access or some change is reflected in the public 


satisfaction level, the findings from the Task Force Pilot 


Project must be, at the least, suspect. 


A number of evaluation issues emerge from the level of 


Service’.area: 


Issue #5 


Tssuerts . 


Issue 


Issue # 


Tssue 


Is there evidence that the Service Bureaux 
Program has resulted in incremental changes 
in the volume of enquiries and changes in 
client satisfaction that would indicate a 
containing need for;"'or modification of 


Program services? 


Are the dynamics of the Service Bureaux 


initiatives supportive of clients' needs: 


— im walk-in offices; 
- in :ltinerant services; and 


“Gan carrels? 


To what extent have the Service Bureaux Program 
initiatives affected the capacity of the special 
target groups to achieve improved levels of 


service (handicapped and disadvantaged) ? 


To wnat extent has the provision of publications 
been. a factor.in providing» improved! services: £6 


the public? 


Is the current limitation of the Service Bureaux 
Program role (e.g. referrals only vs. referrals 

and information/in-person pricri¢cvowss, 1neperson 
and telephone enquiry.) providing.a sSubsicient 

level of service at the testing/experimental 


stage? 
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Issue #10 Has the level of advertising had an impact on 
the Service Bureaux Program operations and to 
what effect? 


2.2.3 Program and Related Dynamics Issues 


The study team devoted some attention to various dynamics 
that could be expected to have significant impact on the 


actual operation. 


Under the resources and management areas the Task Force 
approaches to the training of staff; development of 
procedures and systems and general management of the 
Program were examined. The level of resourcing, in global 
terms was recorded and the views of, field staff were 
obtained on the adequacy and timeliness of Headquarters 


response to their needs were discussed. 


The general view is that, while the Task Force has mounted 

a Significant Program within a pressure-creating time 

frame, the broad issues of training, equipping and directing, 
the staff have been met, albeit by a combination of planned 
and ad hoc decisions and actions. The issues requiring 
examination from a Program Evaluation perspective relate 


more to the continuing, future operation of the Program. 


One issue that tends to stand alone pertains to the use 
of private contract personnel vs. the development of a 


public service organization to provide the staff resources. 


Finally, within this area of dynamics within and without the 
Program, mention should be made of the Federal Departments 
and Agencies, Provincial/Municipal jurisdictions and 
Community Services. Under the terms of reference, the 

study team were unable to explore thesedareas, cinterviews 
with Service Bureaux Program staff indicated a wide 


divergence of views and opinions. As the participation 
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levels, attitudes and roles of these groups are crucial to 


the ultimate provision of service, they must be part of a 


full evaluation. 


The issues within the dynamics area are: 


Issue #ll 


Lesue ai Z 


issue #13 


tesue -34 


Lssue #15 


To what extent and to what effect have 


- Federal Departments and Agencies; 
—- Provincial/Municipal jurisdictions; and 


—- Community Services 
played a role in the Pilot Project initiatives? 


How have internal management, training and 
communications affected the operation of and 
achievements of the Service Bureaux Program 


projects, both regionally and in total? 


Is the level of and use of resourcing available, 
sufficient to achieve the stated objectives of 


the Program? 


Should a Program of providing access/referral to 
the public be manned and operated through Public 
Service staff or through continuing private 

contract arrangements and what are the dynamics, 


benefits and drawbacks of each approach? 


Do the office locations, layouts, image portrayal 
and staff dynamics in handling the public, have a 
bearing on the volumes and types of public use 


anc. to: what erttect. 


The wide range of -issues covered, as indicated earlier, have 


significance for the types of evaluation that could be 


considered by Task Force management. Bx bis. 5 
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has attempted to categorize the issues in terms of the OCG 
concerns and those more related to managements of a Pilot 


Project. 


Section 3, following, considers the Evaluation Options, 


based on the Evaluation Issues we have identified. 


3h PROGRAM EVALUATION OPTIONS 


zs Seg Introduction 
Four basic Program Evaluation Options are presented: 


An Ex-Post Evaluation Option: which would measure 
efficiency and effectiveness to date and relate the findings 
to the potential of the Service Bureau Program to achieve 
stated objectives, including assessments of the effects of 


increasing or decreasing program resource levels. 
An Effectiveness Monitoring Option: wherein a 
determination would be made of the extent to which established 


objectives and desired impacts were achieved. 


A Management Monitoring Option: in which the design, 


process, criteria and data structures and the management 
directives and actions would be evaluated to determine the 


degree to which they contributed to program achievement. 


RK Formative Evaluation Option: aonvolving) intensive 


evaluation of all aspects of the Program to determine the 
revisions needed to ensure the relevancy of objectives and 
processes, the efficient use of resources and designs for 


alternative means of program delivery. 


The above Options are outlined in summary in the following 
sub-sections. An option or Evaluation Approach selected 
by the Task Force would be detailed more extensively in the 


final Evaluation Assessment Report. 
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aie An Ex-Post Evaluation Option 
S24: -SCope 


ine scope of the Ex-Post Evaluation Option is to examine 
the relevancy of the Program's objectives, causal links 
and processes and to measure the efficiency and effective- 


ness of the Program's processes and actions. 
3.2.2 ‘Purpose 
The purpose is to determine: 


- the plausibility of the linkage of objectives 
to activities and effects; 

- the agntended and unintended impacts of the 
Program; 

- the levels of efficiency and effectiveness 
achieved; 

= the*degree’ Or supiicatzon, ‘overlap.or conflict 
with other programs; 

- the effect of discontinuing, increasing or 
decreasing the resource levels; and 

- the possible strengths or weaknesses within 


4 the Program. 


The findings would be used to permit management to decide on 


the future thrusts and’ levels of activity for the Program. 
3.2.3 Methodology 
The methodology, in capsule form would involve: 


- an internal detailing of data resources, 
processes, criteria, performance indicators, 
resourcing and an assessment of each of the 


above elements. 
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—- an extensive survey of client areas to determine 
client Batastaction, levels; 

—- an intensive series of interviews with all 
interacting programs and services; and 

—- detailed cost-effective analysis of alternative 


levels of resourcing. 
3.2.4 Estimated Duration and Study Level 


The study would involve 4 - 5 months of detailed, concentrated 
work, involving a minimum of three consultants and several 
survey/questionnaire. approaches. The estimated cost would 


be in the 50/60 thousand dollar range. 
333 An Effectiveness Monitoring Option 


3.3.4 Scope 


The scope of Effectiveness monitoring Evaluation would be 


to determine to what extent were established objectives and 


desired impacts achieved. 
3.3.2. Purpose 
The purpose of Effectiveness monitoring is to test the: 


—- effectiveness measurement data from Program 
and users; 

- objectives criteria; 

— cost andicators: and 


- other possible impact results. 


The findings would be used to present the current achievement 


levels of the Program and their applicability for the future. 
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32.3.3 Methodology 


The methodology would involve: 


- detailing the Program measures of effectiveness 
and their relation to the objective and desired 
impacts; 

- testing through review of existing data and 
survey of users, the validity of performance 
data; 

- assessing the relative cost effectiveness of 
the results obtained by cost/benefit analysis; and 

- conducting analysis of the proven results against 


Program criteria. 
3.3.4 Estimated Duration and Study Level 
The monitoring of effectiveness would involve 2 to 5 months 
of two consultants and limited survey work. The estimated 
cost would be in the 25/30 thousand dollar range. 
3.4 A Management Monitoring Option 
3.4... Scope 
The scope of this Evaluation Option is narrower and primarily 
targeted at.the efficiency areas of the Program, but with 
an examination of the management policies and practices as 
they relate to Program effectiveness included. 


32.4.2 Purpose 


The purpose is to determine within the accepted objectives, 


causal links and existing processes: 


- the degree to which resources and program 
processes were utilized within planned levels 
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- the extent to which management control and 


direction affected the achievement of results. 


The products of this evaluation option would provide management 
with recommendations for and insights into the Program 


management changes needed to improve direction and control. 
3.4.3. Methodology 
The methodology would encompass: 


= Systems design analysis; 

—- measurement data usage analysis; 

- control mechanism assessment; 

- interviews of staff at all levels; 

- detailed analysis of budgets, costs and 
utilization; and 

- an analytical comparative review of Program 


plans and performance results. 
3.4.4 Estimated Duration and Study Level 


The study of Program Management would involve 2 to 3 months 
of work, involving two consultants, travelling to regional 
bureaux offices for interviews and file/records search and 
Baiveie. The estimated cost would be in the 25/30 thousand 


dollar range. 


< as) A Formative Evaluation Option 


Shee ieCope 


Formative evaluation is an intensive evaluation involving all 
aspects of the Program and stressing and developing alternative 
models for program delivery including reframing of objectives, 
criteria, processes, data base and activities as deemed 


necessary. 
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The purpose of formative evaluation is to determine the 
revisions needed to the Program's objectives, causal 
linkages and processes to ensure their relevancy, the 
efficient use of resources and to ensure the impacts and 


effects meet desired levels. 
3.5.3 Methodology 
The methodology, in brief, would involve: 


—- extensive research of files, documents, 
decisions to analyse rationale and linkage 
of the Program; 

- an intensive detailing of processes, criteria, 
data sources, performance indicators, resourcing 
and assessment of these elements; 

- an intensive survey of users, inter-acting 
agencies and departments to determine 
effectiveness of the Program; 

- examination and analysis of other similar 
programs in Canada and elsewhere to determine 
program delivery designs; and 

—- detailed modelling and cost implication analysis 


to determine effects of proposals. 


3.5.4. Estimated Duration and Study Level 


The study, by its intensive nature would take 5/6 months 
and would involve both consultants and technical design 

experts and extensive survey/interview time and coverage. 
The estimated costs would be in the 60/80 thousand dollar 


range. 
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2:6 A Recommended Evaluation Option 


It should be noted that, in the options presented, no mention 
has been made of advantages and disadvantages under each 


option. 


This approach has been adopted in order that all of the 
options might be discussed within this section. The unique 
and formative nature of the Service Bureaux Program has, in 


our view, dictated this approach. 


Bvaiuataon, Optaions,.,.3 «l and 3). 4 \for -forms;\of .antensive 
evaluation, while perfectly valid options for the Program 
(even in its early stage of development), have disadvantages 
that outweigh the obvious advantages of seriously examining 
the Program and re-orienting it aS early as possible. The 
major and over-riding disadvantage is the lack of sufficient 
performance data and historical context to either support or 
refute Program change at this time. If however, Task Force 
management were to be faced with a deferral on evaluation 
for any period exceeding one year, then either of the 


options should be considered. 


Gption 3.2 which deals with effectiveness-to-date monitoring 
has the distinct advantage of providing senior levels a 
modicum of feed back on which to base decisions on future 
funding and activity levels. The disadvantages of using 
effectiveness monitoring lie in the areas of data reliability 
and applicability and in the relative short life span of 

the Program. This option could be used for evaluation, with 
limited success, within a six month future start-up time 


frame. 


The study team recommends that, if the Program evaluation 
approach is used, the 3.3 Option for a Management Monitoring 
type of evaluation would be the most beneficial. The 


advantages of this option are that it will provide sufficient 
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effectiveness information to meet resource planning needs 
but the main focus would be improvements within accepted 
current objectives and structure. The disadvantages tend 

tor ret tec, the constant (for'ieach option and relate to the 
limited data and time of existence of the Program. These, 
however, are less significant under the Management Monitoring 


Evaluation Option. 


Each of the options presented, including the preferred 
options, would deal, to a greater or lesser degree with 
all 15 evaluation issues. The detail of the extent to which 
each issue would be explored will be highlighted in the 


option selected by management. 


a A Project Evaluation Alternative Approach 


The question of determining between "Program" and) "Project" 
evaluation is not merely a semantic problem. 
Program Evaluation, by definition embraces two key 


fundamentals that must be explored definitively. 


- Program Rationale, and 


- Program Alternatives. 


The sttdy team do not believe that either of these areas 
require exploration at this point in the life of the Service 


Bureaux Program. 


Conversley Program Evaluation, by definition avoids an 
overlap with management audit concerns on systems and 


procedures in a compliance sense. 


In this area the study teams believes some elements of 
management's practices and resourceful meeting of objectives 
should be reflected in studying the challenge and response 
to the bureaucratic processes, normally reported through 


management audit studies. 
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Within a Project Evaluation the rationale for the Program 
is accepted and similarily alternatives to the current 
Program design and delivery are produced as by-products 


rather than a focused GETOxrt. 


Project Evaluation would provide products directly related 
to the present and future management of the Pilot Projects 


Such questions as: 


~ ine Suatability Of the design and location of 
offices; 

the effect-oL Signage and amage! portrayed 
through furnishings etc; 

- the start approaches to various publics; 

— tne Contract start VS. public service staff: 

- the organizational issues affecting the Program; 
plus 

- the range of evaluation issues normally associated 
with "Objectives Achievement" and "Impacts and 


Effects" under program evaluation. 


The freedom to design the evaluation to more immediate 
needs of management in resource planning, internal process/ 
data adjustments and managerial improvements, seems better 


Suited to a Program that: 


- has recently been introduced; 

- has a varied range of approaches; 

- is designed to provide answers, but also provide 
service; and 

- has the vitality. and ability to adapt, anherent in 
the Pilot Project Approach. 


The obvious disadvantage is that the weight of approval in 
approach from the Office of the Comptroller General would be 


missing. The study team believes very strongly however that, 
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within Program Evaluation Concepts we should declare the 


Program as not evaluable, given its embryo status. 


Through Project Evaluation it is possible to develop, along 
with evaluation findings, an Evaluation Framework for a 


future, more intensive Program Evaluation. 


The duration of the proposed Project Evaluation would be 


3/4 months, at: an estimated range wor 307 S5sethousand dollars. 
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